Skip to content

wandb/senpai / .agents/skills/analyze-experiments

analyze-experiments

Verified

Operator-side analysis of historical ML experiment PRs in Senpai research tracks. Use this skill whenever the user asks to: analyze experiments, categorize PRs, bucket experiments, summarize what's been tried, understand experiment history, review merged vs closed results, or asks "what experiments have we run / worked / failed". Also triggers for: "pull the latest experiments", "what's been tried so far", "category breakdown of PRs", "which experiments succeeded", "noam track analysis". When a branch name is mentioned (e.g. "noam branch", "on the noam branch"), pass it as the base branch to scope the fetch to just those PRs.

Analyze Experiments Skill

This is a human/operator analysis guide, not a live advisor or student skill. It fetches fresh experiment PR data via the list-experiments skill, categorizes each PR using parallel readers, and produces a 5-section report: full catalogue, category breakdown with merge rates, merged-only wins, closed-only failures, and key narratives.


Step 1: Fetch PR data via list-experiments

Read the list-experiments skill at .agents/skills/list-experiments/SKILL.md and run its Python code with the following configuration:

If a base branch was specified (e.g. "noam", "yan", "main") — use it directly:

BASE_BRANCH = "noam"   # replace with the branch the user mentioned
# use the fetch command from list-experiments as-is (it uses --base BASE_BRANCH)

If no branch was specified — fetch all senpai-labelled experiments instead. Replace the fetch() function body with:

cmd = ["gh", "pr", "list", "--repo", "wandb/senpai", "--json", FIELDS,
       "--limit", "10000", "--state", "all", "--label", "senpai"]

Run the script and note the experiments_summary_<ts>.md path printed — that's your input for Steps 2–3.


Step 2: Find split points for parallel reading

The summary file can be large (thousands of lines). Divide it into 4 roughly equal chunks:

# Get total PR count and line positions for every ~25% boundary
grep -n "^# PR" <summary_path> | awk 'BEGIN{getline l1; print l1} NR==int(total*0.25) || NR==int(total*0.5) || NR==int(total*0.75) {print} END{print}' total=$(grep -c "^# PR" <summary_path>)

# Simpler: just get all PR line numbers and pick 3 evenly-spaced split points
grep -n "^# PR" <summary_path> | awk -v n=$(grep -c "^# PR" <summary_path>) 'NR==1||NR==int(n/4)||NR==int(n/2)||NR==int(3*n/4)||NR==n{print NR, $0}'

Note the line numbers for the 4 batch boundaries.


Step 3: Launch 4 parallel Explore agents

Send all 4 in a single message. Each agent reads its assigned line range from the summary file and returns one row per PR. The agents need to read both the title AND the ## Results section to assign an accurate outcome — title-based keywords set the category, the results section determines whether it worked.

Give each agent these instructions (substituting LINE_START, LINE_END, PR range):

Read <summary_path> from line LINE_START to LINE_END. For every PR in this range return exactly one row: PR #NNN | STATE | Category | emoji | 1-line outcome

STATE: MERGED / CLOSED / OPEN (from the | State | header table in each PR block)

Category — pick exactly one based on the primary change being tested:

  1. Loss function — loss type (L1, MSE, Huber, cosine sim, asymmetric surf/vol split)
  2. LR / optimizer — learning rate value, warmup, scheduler, weight decay, β params, optimizer choice
  3. Model architecture — depth (n_layers), width (n_hidden), n_heads, mlp_ratio, output head design, preprocess MLP structure
  4. Initialization — weight init strategy (Xavier, Kaiming, orthogonal, learnable placeholders, init scale/gain)
  5. Training efficiency — bf16 autocast, batch size, gradient accumulation, volume node subsampling, epoch budget tricks
  6. Regularization — dropout, SWA, EMA, target noise magnitude/schedule, gradient clipping, spectral norm
  7. Physics / normalization — Cp normalization, per-sample normalization, domain-aware scaling, split surf/vol stats
  8. Loss weighting — surf_weight value or schedule, per-channel weights, domain re-weighting
  9. Feature engineering — slice count/structure, attention temperature, spatial/positional encoding (RFF, NeRF)
  10. Inference fix — fp32 for OOD splits, NaN guards, denormalization fixes, clamping

emoji: ✓ improved primary metric (mae_surf_p on val_in_dist), ✗ hurt it, ~ inconclusive/noise-level, — no results yet

1-line outcome: state what changed, the key metric delta (e.g. "mae_surf_p 119→103 (-14%)"), and the reason it succeeded or failed. If Results section is empty, write "No results".

Return ONLY the data rows — no headers, no commentary.


Step 4: Synthesize into report

Combine all rows and produce a 5-section markdown report:

Section 1 — Full PR Catalogue Table: | PR | State | Category | Result | Outcome | All PRs sorted by number. OPEN PRs included but marked —.

Section 2 — Category Breakdown (completed PRs only) Table: | Category | Total | Merged | Closed | Merge Rate | Merged PRs | Compute merge rate = MERGED ÷ (MERGED + CLOSED). Sort descending by merge rate. OPEN PRs excluded from totals and rates.

Section 3 — Merged PRs Only For each category that has merges: list the merged PRs and a sentence on what the wins had in common.

Section 4 — Closed PRs — Failure Patterns Table: | Category | Closed | Hard ✗ | Inconclusive ~ | Showed promise ✓ (not merged) | Call out which categories had PRs that showed improvement but still weren't merged.

Section 5 — Key Narratives 3–5 bullet points on cross-cutting patterns: what reliably works, what never works, what the current productive frontier looks like (recent OPEN PRs).


Notes

  • OPEN PRs: include in catalogue with — outcome; exclude from all merge-rate calculations.
  • Primary metric: mae_surf_p on val_in_dist. When that's unavailable use any reported split. Lower is better.
  • Metric era shift: Cp normalization (around PR #392 in the noam track) changed mae_surf_p from ~80–200 Pa to ~20–50 Pa range. Don't compare raw numbers across this boundary — only relative improvement within each era matters.
  • Initialization vs Model architecture: If a PR changes only the init strategy (Xavier, orthogonal, Kaiming, learnable init), use Initialization. If it changes both init and something structural, use whichever is the primary tested hypothesis.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall