wandb/senpai / .agents/skills/analyze-experiments
analyze-experiments
VerifiedOperator-side analysis of historical ML experiment PRs in Senpai research tracks. Use this skill whenever the user asks to: analyze experiments, categorize PRs, bucket experiments, summarize what's been tried, understand experiment history, review merged vs closed results, or asks "what experiments have we run / worked / failed". Also triggers for: "pull the latest experiments", "what's been tried so far", "category breakdown of PRs", "which experiments succeeded", "noam track analysis". When a branch name is mentioned (e.g. "noam branch", "on the noam branch"), pass it as the base branch to scope the fetch to just those PRs.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.5 KB |
| evals/evals.json | config | 4.4 KB |
evals/evals.json
View on GitHub{
"skill_name": "analyze-experiments",
"evals": [
{
"id": 0,
"prompt": "pull the latest noam branch experiments and give me a full category breakdown — what's been tried, what merged, what closed",
"expected_output": "A markdown report with: (1) full PR catalogue table, (2) category breakdown table with merge rates, (3) merged-only summary, (4) closed-only failure patterns, (5) key narrative insights. Should use parallel subagents to analyze the PRs.",
"files": [],
"assertions": [
{
"name": "uses-9-category-taxonomy",
"description": "Report uses the exact 9 categories: Loss function, LR / optimizer, Model architecture, Training efficiency, Regularization, Physics / normalization, Loss weighting, Feature engineering, Inference fix — at least 7 of 9 must appear",
"type": "content_check"
},
{
"name": "has-three-views",
"description": "Report contains all three views: all PRs, merged-only, and closed-only sections",
"type": "content_check"
},
{
"name": "includes-merge-rates",
"description": "Category breakdown table includes merge rate percentages",
"type": "content_check"
},
{
"name": "includes-pr-catalogue",
"description": "Report includes a full PR catalogue table with individual PR rows (not just summaries)",
"type": "content_check"
},
{
"name": "identifies-cp-normalization-as-step-change",
"description": "Report identifies PR #392 (Cp normalization) as a major/step-change improvement",
"type": "content_check"
},
{
"name": "has-narrative-insights",
"description": "Report includes a narrative insights or key findings section with cross-cutting patterns",
"type": "content_check"
},
{
"name": "report-length",
"description": "Report is at least 1000 words (comprehensive analysis of 150 PRs)",
"type": "length_check"
}
]
},
{
"id": 1,
"prompt": "i want to understand which experiment categories have had the best success rate in the senpai project. fetch the latest data and categorize all the PRs",
"expected_output": "A category breakdown showing merge rates per category, with the full PR catalogue and identification of which categories produced the most merged PRs.",
"files": [],
"assertions": [
{
"name": "uses-9-category-taxonomy",
"description": "Report uses the exact 9 categories from the skill taxonomy — at least 7 must appear",
"type": "content_check"
},
{
"name": "has-merge-rate-comparison",
"description": "Report ranks or compares categories by success/merge rate",
"type": "content_check"
},
{
"name": "includes-pr-catalogue",
"description": "Report includes individual PR entries, not just aggregate counts",
"type": "content_check"
},
{
"name": "fetches-fresh-data",
"description": "Report mentions fetching data or running a script (not just working from memory)",
"type": "content_check"
}
]
},
{
"id": 2,
"prompt": "bucket the experiment prs — show me all, then just the merged ones, then just the failed ones",
"expected_output": "Three views of the experiment PRs: all PRs categorized, merged-only view, closed-only failure patterns view. Categories from the 9-category taxonomy.",
"files": [],
"assertions": [
{
"name": "has-all-prs-section",
"description": "Report has a section showing all PRs (or a full catalogue)",
"type": "content_check"
},
{
"name": "has-merged-only-section",
"description": "Report has a distinct section for merged/successful PRs only",
"type": "content_check"
},
{
"name": "has-closed-only-section",
"description": "Report has a distinct section for closed/failed PRs only",
"type": "content_check"
},
{
"name": "uses-consistent-categories",
"description": "All three views use consistent category names from the 9-category taxonomy",
"type": "content_check"
}
]
}
]
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |