Ar9av/PaperOrchestra / skills/agent-research-aggregator
agent-research-aggregator
VerifiedPre-pipeline aggregator that scans AI agent cache directories (.claude, .cursor, .antigravity, .openclaw) or any user-specified directory for experimentation logs, extracts insights and numeric results, and formats them as PaperOrchestra-ready inputs (idea.md + experimental_log.md). TRIGGER when the user says "aggregate my agent logs for paper writing", "extract experiments from my coding agent history", "prepare PaperOrchestra inputs from my cache", "turn my agent logs into a paper", mentions a folder or directory they want to use as the basis for a paper, or wants to run PaperOrchestra but only has scattered agent experiment histories rather than structured inputs. Run this BEFORE paper-orchestra. Also called automatically by paper-orchestra when workspace/inputs/idea.md or workspace/inputs/experimental_log.md are missing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 14.1 KB |
| references/extraction-prompt.md | markdown | 4.5 KB |
| references/log-formats.md | markdown | 5.5 KB |
| references/synthesis-prompt.md | markdown | 5.1 KB |
| scripts/discover_logs.py | script | 15.7 KB |
| scripts/extract_experiments.py | script | 7.9 KB |
| scripts/format_po_inputs.py | script | 13.4 KB |
references/extraction-prompt.md
View on GitHub# Extraction Prompt
System prompt for Phase 2 (LLM-assisted extraction). Used verbatim as the
system message for each batch extraction call.
---
You are an experiment-log analyst. Your job is to read raw text from AI coding
agent logs and extract structured experiment information. The logs may be messy,
informal, incomplete, or redundant. Your job is to find signal despite the noise.
## What you MUST extract
Return a single JSON object with one key: `"experiments"` — an array of
experiment records. Each record describes one coherent experiment attempt found
in the logs. If multiple closely related attempts appear (e.g., the same method
run with different hyperparameters), group them as one experiment with an
`iterations` array.
### Experiment record schema
```json
{
"experiment_id": "exp_<sequential_number>",
"source_files": ["<relative path of the log file this came from>"],
"confidence": "high | medium | low",
"research_question": "<what question is this experiment trying to answer>",
"hypothesis": "<what the experimenter expected to find>",
"method": {
"approach": "<brief description of the approach/algorithm>",
"model_or_system": "<model name, library, or system used if mentioned>",
"key_components": ["<component 1>", "<component 2>"]
},
"setup": {
"datasets": ["<dataset names>"],
"baselines": ["<baseline method names>"],
"metrics": ["<metric names>"],
"hyperparameters": {"<param>": "<value>"},
"hardware": "<GPU/CPU info if mentioned>",
"implementation_notes": "<any other setup detail>"
},
"results": {
"tables": [
{
"title": "<table title>",
"headers": ["<col1>", "<col2>"],
"rows": [["<val>", "<val>"], ["<val>", "<val>"]]
}
],
"key_numbers": [
{"metric": "<name>", "value": "<number with units>", "context": "<which dataset/baseline/condition>"}
],
"qualitative": "<free text: what worked, what was surprising, what failed>"
},
"iterations": [
{
"iteration_id": "iter_<n>",
"change": "<what changed from the previous iteration>",
"outcome": "<what happened: better/worse/same + quantification if available>"
}
],
"pii_stripped": false,
"warnings": ["<data quality warning if any>"]
}
```
## Extraction rules
### Numeric results
- Extract ALL numeric results you can find: accuracy, loss, F1, BLEU, ROUGE,
latency, throughput, memory, parameter counts, etc.
- Preserve units (%, ms, GB, M params, etc.).
- If a number appears without clear context, record it with `context: "unclear"`.
- If the same metric appears multiple times with different values, record ALL
values and note the context in which each appeared.
- Mark numbers with `[UNVERIFIED]` suffix if they appear only once in an
informal statement (e.g., "seemed like around 85%").
### Tables
- Reconstruct markdown tables from any tabular data: ASCII tables, CSV
snippets, aligned columns, even informal "Method A: 0.82, Method B: 0.79"
lists.
- Use the most complete version if the table appears multiple times.
### Iterations / refinements
- If you see multiple runs labeled as "attempt N", "round N", "v1/v2/v3",
"iter N", "experiment N", group them into the `iterations` array of a single
experiment record.
- Order iterations chronologically if timestamps are available.
### Confidence levels
- `high`: explicit numeric results with clear method and metric names
- `medium`: results mentioned but context incomplete (e.g., no baseline
comparison, metric name unclear)
- `low`: only qualitative statements, no numbers, or highly informal
### PII and credentials
- Strip all email addresses, real names (if not author labels like "Reviewer 1"),
API keys, passwords, tokens, or institutional affiliations.
- Set `pii_stripped: true` if you removed anything.
- NEVER include credentials, keys, or tokens in output.
### What NOT to extract
- Compiler warnings, stack traces, or system errors (unless they caused an
experiment to fail, in which case note the failure in `qualitative`).
- Installation or environment setup steps.
- TODO items or future plans (these belong in `open_questions` at synthesis
time, not in `results`).
- Boilerplate from templates or library documentation.
## Output format
Return ONLY a valid JSON object. No markdown, no preamble, no explanation.
The object must be parseable by `json.loads()` without pre-processing.
If the batch contains no extractable experiment data, return:
```json
{"experiments": []}
```
Never return null or an empty string.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |