wandb/senpai / .agents/skills/experiment-report
experiment-report
VerifiedCreate consistent nn_cfd W&B experiment comparison reports from selected run IDs using wandb and wandb_workspaces. Use this whenever the user asks to report experiment results, create a W&B Report, compare CFD training/eval runs, document baseline-vs-variant outcomes, reproduce the project experiment report template, or publish a leaderboard-style report for milieu/nn_cfd.
Experiment Report
Use this skill to turn selected nn_cfd W&B runs into the project-standard
experiment comparison report. The canonical template is the W&B report
[DO NOT DELETE/EDIT] Template Experiment Comparison Report.
Workflow
- Identify the baseline run first. If the user did not provide it, ask for the baseline run ID or run name before creating the report.
- Identify the experiment run IDs. Prefer exact run IDs over names, groups, or regex searches.
- Query W&B summaries and configs with the W&B SDK. Use bounded metric keys; do not scan broad histories unless the report needs a curve-specific claim.
- Read
references/template.mdbefore making a report by hand. It records the report width, runset settings, panel positions, plot axis limits, media keys, and expected block order from the template. - Prefer
scripts/create_experiment_report.pyfor consistency. It builds the report from run IDs, validates the summary metrics needed by the table, and recreates the template panel grid. - Always publish the report. Do not leave experiment reports as W&B drafts.
- Name every report
[YYYY-MM-DD] NAME, where the date is the report's first creation date. Preserve that date on later edits because the W&B Reports UI shows the last modified date, not the original creation date.
Report Command
Run from the repo root:
uv run python .agents/skills/experiment-report/scripts/create_experiment_report.py \
--baseline_run_id a9300aedc8926440 \
--experiment_run_ids 854f115d43294ece qz4xm8cq ch930ho8 \
--title "cape domain residual variants"
Useful optional arguments:
--human_result: fills the TL;DR block. If absent, keep the template's human placeholder.--hypothesis: fills the hypothesis block. If absent, keep the template placeholder.--core_changes: describes the tested changes in Experiment Setup.--pr_url: adds the PR link.--report_date: pins the[YYYY-MM-DD]prefix when recreating or updating a report whose first-created date is not today.--created_at_max: adds the template-style CreatedTimestamp runset cap when cloning a historical report snapshot. Prefer exact run IDs without this cap for normal reports.
Report Standards
- Use
wandb_workspaces.reports.v2, not deprecatedwandb.apis.reports. - Use
width="fluid"so reports render full-width. The report builder must force this width immediately before save/upsert and verify the saved report model spec still haswidth="fluid"; do not silently accept W&B's defaultreadablewidth. CheckReport.from_url(url, as_model=True).spec.widthbecause the hydratedReportobject can fall back to its local default. - Use
report.save(draft=False); this project does not keep hanging draft experiment reports. - Use report titles of the form
[YYYY-MM-DD] NAME. The script will add today's date if--titleomits the bracketed prefix. - Filter the runset by exact run ID with
expr.Metric("name").isin(run_ids). The W&B UI may display this asMetric("ID"), but the structured expression should use lowercasename. - Sort by
CreatedTimestampascending. - Hide the
run:namecolumn. - Set
RunSettings(disabled=True)for the baseline by default, matching the template's plot behavior, while still keeping the baseline in the markdown results table. - Use
wr.MarkdownBlockfor tables and markdown headings. Do not put markdown tables insidewr.P. - In Experiment Setup, render
Input featuresandTargetsas separate YAML blocks. Never bury target fields inside an input/features block or say that targets are "included above." - Keep the Results table to two decimal places and include run links, run IDs, and validation-set labels.
- In the Results table, every run name must be a plain markdown link to its W&B run URL. Escape square brackets in run names before using them as link labels, and do not rely on unlabeled IDs or placeholder text.
- State missing evidence plainly. Do not invent PR links, hypotheses, or human final verdicts.
Resources
scripts/create_experiment_report.py: deterministic report builder for the template.references/template.md: exact template anatomy and panel layout.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |