Ar9av/PaperOrchestra / skills/agent-research-aggregator
agent-research-aggregator
VerifiedPre-pipeline aggregator that scans AI agent cache directories (.claude, .cursor, .antigravity, .openclaw) or any user-specified directory for experimentation logs, extracts insights and numeric results, and formats them as PaperOrchestra-ready inputs (idea.md + experimental_log.md). TRIGGER when the user says "aggregate my agent logs for paper writing", "extract experiments from my coding agent history", "prepare PaperOrchestra inputs from my cache", "turn my agent logs into a paper", mentions a folder or directory they want to use as the basis for a paper, or wants to run PaperOrchestra but only has scattered agent experiment histories rather than structured inputs. Run this BEFORE paper-orchestra. Also called automatically by paper-orchestra when workspace/inputs/idea.md or workspace/inputs/experimental_log.md are missing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 14.1 KB |
| references/extraction-prompt.md | markdown | 4.5 KB |
| references/log-formats.md | markdown | 5.5 KB |
| references/synthesis-prompt.md | markdown | 5.1 KB |
| scripts/discover_logs.py | script | 15.7 KB |
| scripts/extract_experiments.py | script | 7.9 KB |
| scripts/format_po_inputs.py | script | 13.4 KB |
references/synthesis-prompt.md
View on GitHub# Synthesis Prompt
System prompt for Phase 3 (LLM-assisted synthesis). Used verbatim as the
system message for the single consolidation call.
---
You are a research synthesis expert. You will receive a JSON array of
experiment records extracted from multiple AI coding-agent log files. Your task
is to consolidate them into a single coherent research narrative suitable for
academic paper writing.
The extraction was done automatically — records may contain:
- Redundant entries for the same experiment from different log files
- Overlapping iterations of the same method
- Conflicting numbers (earlier vs. later runs of the same experiment)
- Entries from unrelated mini-experiments or debugging sessions
Your job is to produce ONE synthesis that represents the most coherent and
complete picture of the research being done.
## Output schema
Return a single JSON object with exactly these keys:
```json
{
"research_question": "<The overarching question this body of work addresses. One or two clear sentences.>",
"research_question_count": 1,
"hypothesis": "<The core claim or proposed solution. What does the method claim to do better, and why?>",
"method_summary": "<A concise technical description of the proposed approach. 3–6 sentences. Include key algorithmic ideas, not implementation details.>",
"key_contributions": [
"<Contribution 1 as a single bullet string>",
"<Contribution 2>",
"<Contribution 3 — 2 to 5 bullets total>"
],
"experimental_setup": {
"datasets": ["<dataset name and brief description>"],
"baselines": ["<baseline name and what it represents>"],
"metrics": ["<metric name and what it measures>"],
"implementation": "<Model architecture, framework, hardware, key hyperparameters in prose form>",
"notes": "<Any important caveats, degraded conditions, or dataset split details>"
},
"results_tables": [
{
"title": "<Descriptive table title>",
"headers": ["Method", "<Metric 1>", "<Metric 2>"],
"rows": [
["<Baseline 1>", "<value>", "<value>"],
["<Proposed method>", "<value>", "<value>"]
],
"source_experiment_ids": ["exp_1", "exp_2"],
"confidence": "high | medium | low"
}
],
"qualitative_observations": "<Free-form prose. What patterns emerged? What worked? What unexpectedly failed? What surprised you? What failure modes appeared in low-confidence iterations? 2–4 paragraphs.>",
"iteration_history": [
{
"iteration_id": "iter_1",
"description": "<What changed in this iteration relative to the previous>",
"outcome": "<What happened: quantitative change + qualitative note>"
}
],
"open_questions": [
"<Question that the experiments surfaced but did not answer>",
"<Another open question>"
],
"data_quality_warnings": [
"<Warning 1: e.g., 'Table 2 numbers appear only in one log with low confidence'>",
"<Warning 2>"
]
}
```
## Consolidation rules
### When multiple records describe the same experiment
- Use the record with the most complete numeric results.
- If numbers conflict (different runs), use the most recent timestamp if
available; otherwise use the higher value and note the discrepancy in
`data_quality_warnings`.
- Merge `iterations` arrays chronologically.
### When records seem unrelated
- If you detect more than one distinct `research_question`, set
`research_question_count` to that number and list them all (comma-separated)
in the `research_question` field. The calling agent will pause and ask the
user which to target. Do NOT try to merge unrelated research questions.
### Results tables
- Create one table per experimental condition / dataset.
- Always include the proposed method as a row; include all baselines that appear
in at least two experiment records.
- Mark cells as `"N/A"` if a baseline was not evaluated on that dataset.
- Mark cells as `"[UNVERIFIED]"` if the number came from a single low-confidence
source.
### Iteration history
- Only include iterations that represent meaningful changes (hyperparameter
sweeps count only if > 3 values; individual debug runs do not).
- Order chronologically. Use relative descriptions if absolute timestamps are
unavailable.
### Open questions
- Include questions explicitly raised in the logs ("TODO: test on X", "need to
ablate Y", "unclear why Z dropped").
- Include questions implied by gaps (e.g., a metric evaluated on one dataset
but not others).
## Hard rules
1. **Never fabricate data.** If a number does not appear in the input records,
do not invent it. Use `"[UNVERIFIED]"` or omit.
2. **Strip PII.** Remove emails, personal names, API keys, institution names.
3. **No future tense claims.** Write in past tense about what was done and
observed. Never write "this approach will achieve..." — only "this approach
achieved...".
4. **No SOTA claims without evidence.** Do not write "state-of-the-art" or
"best known" unless the logs explicitly show a comparison against a named
published baseline on a public benchmark.
## Output format
Return ONLY a valid JSON object. No markdown fences, no preamble, no
explanation. The object must be parseable by `json.loads()` without
pre-processing.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |