Ar9av/PaperOrchestra / skills/agent-research-aggregator
agent-research-aggregator
VerifiedPre-pipeline aggregator that scans AI agent cache directories (.claude, .cursor, .antigravity, .openclaw) or any user-specified directory for experimentation logs, extracts insights and numeric results, and formats them as PaperOrchestra-ready inputs (idea.md + experimental_log.md). TRIGGER when the user says "aggregate my agent logs for paper writing", "extract experiments from my coding agent history", "prepare PaperOrchestra inputs from my cache", "turn my agent logs into a paper", mentions a folder or directory they want to use as the basis for a paper, or wants to run PaperOrchestra but only has scattered agent experiment histories rather than structured inputs. Run this BEFORE paper-orchestra. Also called automatically by paper-orchestra when workspace/inputs/idea.md or workspace/inputs/experimental_log.md are missing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 14.1 KB |
| references/extraction-prompt.md | markdown | 4.5 KB |
| references/log-formats.md | markdown | 5.5 KB |
| references/synthesis-prompt.md | markdown | 5.1 KB |
| scripts/discover_logs.py | script | 15.7 KB |
| scripts/extract_experiments.py | script | 7.9 KB |
| scripts/format_po_inputs.py | script | 13.4 KB |
scripts/extract_experiments.py
View on GitHub#!/usr/bin/env python3
"""
extract_experiments.py — Phase 2 helper for agent-research-aggregator.
This script has two modes:
1. BATCH MODE (called by the host agent to prepare batches):
Reads discovered_logs.json, groups files into batches under a size budget,
and prints the list of file paths per batch so the host LLM knows what to
read and extract.
python extract_experiments.py \\
--discovered workspace/ara/discovered_logs.json \\
--list-batches \\
--batch-bytes 40000
2. VALIDATE MODE (called after host has done all LLM extraction calls):
Reads raw_experiments.json produced by the host and validates it meets
the minimum schema before Phase 3.
python extract_experiments.py \\
--discovered workspace/ara/discovered_logs.json \\
--out workspace/ara/raw_experiments.json \\
--validate-only
Usage for validate mode:
python extract_experiments.py \\
--out workspace/ara/raw_experiments.json \\
--validate-only
"""
import argparse
import json
import sys
from pathlib import Path
REQUIRED_TOP_KEYS = {"experiments"}
EXPERIMENT_REQUIRED = {"experiment_id", "confidence"}
EXPERIMENT_ONE_OF = {"hypothesis", "method", "results", "research_question"}
VALID_CONFIDENCE = {"high", "medium", "low"}
# ---------------------------------------------------------------------------
# Batch listing (for host agent to know what to read)
# ---------------------------------------------------------------------------
def list_batches(discovered_path: str, batch_bytes: int):
with open(discovered_path, encoding="utf-8") as f:
manifest = json.load(f)
files = manifest.get("files", [])
if not files:
print("No files in manifest.", file=sys.stderr)
sys.exit(1)
batches: list[list[dict]] = []
current_batch: list[dict] = []
current_size = 0
for entry in files:
size = min(entry["size_bytes"], 200 * 1024) # cap at truncation limit
if current_batch and current_size + size > batch_bytes:
batches.append(current_batch)
current_batch = []
current_size = 0
current_batch.append(entry)
current_size += size
if current_batch:
batches.append(current_batch)
print(f"Total batches: {len(batches)}")
print(f"Total files : {len(files)}")
print()
for i, batch in enumerate(batches, 1):
total = sum(min(e["size_bytes"], 200*1024) for e in batch)
print(f"--- Batch {i} ({len(batch)} files, ~{total//1024} KB) ---")
for entry in batch:
trunc = " [TRUNCATED]" if entry.get("truncated") else ""
print(f" [{entry['priority']:6}] [{entry['agent']:12}] {entry['path']}{trunc}")
print()
# ---------------------------------------------------------------------------
# Validation
# ---------------------------------------------------------------------------
def validate_experiments(out_path: str) -> bool:
path = Path(out_path)
if not path.exists():
print(f"[ERROR] File not found: {out_path}", file=sys.stderr)
return False
try:
with path.open(encoding="utf-8") as f:
data = json.load(f)
except json.JSONDecodeError as e:
print(f"[ERROR] Invalid JSON: {e}", file=sys.stderr)
return False
# Top-level structure
missing_top = REQUIRED_TOP_KEYS - set(data.keys())
if missing_top:
print(f"[ERROR] Missing top-level keys: {missing_top}", file=sys.stderr)
return False
experiments = data["experiments"]
if not isinstance(experiments, list):
print("[ERROR] 'experiments' must be a list", file=sys.stderr)
return False
if len(experiments) == 0:
print("[WARN] 'experiments' array is empty — no extractable data found.")
print("This may be correct if logs contained no experiment data.")
print("Proceeding to Phase 3 with empty input is allowed but will")
print("produce a synthesis with no results — confirm with user.")
errors = []
warnings = []
for i, exp in enumerate(experiments):
label = exp.get("experiment_id", f"[index {i}]")
# Required keys
for key in EXPERIMENT_REQUIRED:
if key not in exp:
errors.append(f"{label}: missing required key '{key}'")
# At least one of these must be present
if not any(k in exp for k in EXPERIMENT_ONE_OF):
errors.append(f"{label}: must have at least one of {EXPERIMENT_ONE_OF}")
# Confidence value
conf = exp.get("confidence", "")
if conf not in VALID_CONFIDENCE:
errors.append(f"{label}: 'confidence' must be one of {VALID_CONFIDENCE}, got '{conf}'")
# Results tables shape
results = exp.get("results", {})
if isinstance(results, dict):
for j, table in enumerate(results.get("tables", [])):
if not isinstance(table.get("headers"), list):
errors.append(f"{label}: results.tables[{j}].headers must be a list")
if not isinstance(table.get("rows"), list):
errors.append(f"{label}: results.tables[{j}].rows must be a list")
# Warn about low-confidence experiments with no numeric data
if conf == "low":
key_nums = results.get("key_numbers", []) if isinstance(results, dict) else []
tables = results.get("tables", []) if isinstance(results, dict) else []
if not key_nums and not tables:
warnings.append(f"{label}: low confidence + no numeric data")
for w in warnings:
print(f"[WARN] {w}")
if errors:
for e in errors:
print(f"[ERROR] {e}", file=sys.stderr)
print(f"\nValidation FAILED: {len(errors)} error(s), {len(warnings)} warning(s)")
return False
print(f"Validation PASSED: {len(experiments)} experiment(s), {len(warnings)} warning(s)")
high = sum(1 for e in experiments if e.get("confidence") == "high")
med = sum(1 for e in experiments if e.get("confidence") == "medium")
low = sum(1 for e in experiments if e.get("confidence") == "low")
print(f" Confidence: {high} high / {med} medium / {low} low")
tables_total = sum(
len(e.get("results", {}).get("tables", []))
for e in experiments if isinstance(e.get("results"), dict)
)
print(f" Result tables found: {tables_total}")
return True
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
parser = argparse.ArgumentParser(description="Phase 2 helper: batch listing and validation")
parser.add_argument("--discovered", default=None,
help="Path to discovered_logs.json (required for --list-batches)")
parser.add_argument("--out", default=None,
help="Path to raw_experiments.json (required for --validate-only)")
parser.add_argument("--list-batches", action="store_true",
help="Print batches of files for the host agent to process")
parser.add_argument("--batch-bytes", type=int, default=40000,
help="Soft byte budget per LLM extraction batch (default: 40000)")
parser.add_argument("--validate-only", action="store_true",
help="Validate raw_experiments.json (requires --out)")
args = parser.parse_args()
if args.list_batches:
if not args.discovered:
print("[ERROR] --list-batches requires --discovered", file=sys.stderr)
sys.exit(1)
list_batches(args.discovered, args.batch_bytes)
sys.exit(0)
if args.validate_only:
if not args.out:
print("[ERROR] --validate-only requires --out", file=sys.stderr)
sys.exit(1)
ok = validate_experiments(args.out)
sys.exit(0 if ok else 1)
parser.print_help()
sys.exit(1)
if __name__ == "__main__":
main()
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |