wandb/senpai / plugins/senpai/skills/wandb-primary
wandb-primary
VerifiedComprehensive primary skill for agents working with Weights & Biases. Covers both the W&B SDK (training runs, metrics, artifacts, sweeps) and the Weave SDK (GenAI traces, evaluations, scorers). Includes helper libraries, gotcha tables, and data analysis patterns. Use this skill whenever the user asks about W&B runs, Weave traces, evaluations, training metrics, loss curves, model comparisons, or any Weights & Biases data — even if they don't say "W&B" explicitly.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 16.6 KB |
| references/TRAINING_DIAGNOSTICS.md | markdown | 6.6 KB |
| references/WANDB_SDK.md | markdown | 12.9 KB |
| references/WEAVE_SDK.md | markdown | 6.6 KB |
| scripts/curve_plots.py | script | 16.1 KB |
| scripts/step_axis.py | script | 5.8 KB |
| scripts/training_diagnostics.py | script | 18.6 KB |
| scripts/wandb_helpers.py | script | 7.7 KB |
| scripts/weave_helpers.py | script | 12.4 KB |
SKILL.md
View on GitHub---
# SPDX-FileCopyrightText: 2026 CoreWeave, Inc.
# SPDX-License-Identifier: Apache-2.0
# SPDX-PackageName: skills
name: wandb-primary
description: Comprehensive primary skill for agents working with Weights & Biases. Covers both the W&B SDK (training runs, metrics, artifacts, sweeps) and the Weave SDK (GenAI traces, evaluations, scorers). Includes helper libraries, gotcha tables, and data analysis patterns. Use this skill whenever the user asks about W&B runs, Weave traces, evaluations, training metrics, loss curves, model comparisons, or any Weights & Biases data — even if they don't say "W&B" explicitly.
---
# W&B Primary Skill
## Python environment
Senpai configures `python` and uv to use the writable target environment.
Run analysis scripts with the installed package set:
```bash
uv run --no-sync python analysis.py
```
`--no-sync` prevents an analysis command from resolving or syncing the target
project's dependencies. For additive installs that reuse shared image packages,
use `python -m pip install PACKAGE`. Use `uv pip install --no-deps PACKAGE` when
the package's dependencies are already available. The pinned uv 0.10.9 resolver
does not inspect the shared image package path; `uv pip install` can install
duplicate dependencies, including large CUDA packages. Use `uv add` or `uv sync`
when you intend to update or sync the target project's separate dependency set.
Keep analysis scripts and outputs in the target workspace or `/tmp`. The bundled
skill and helper files under `$SENPAI_PLUGIN` are read-only; do not edit them.
Plot helpers default to `/tmp/wandb_plots`; pass `out_dir` to select another
writable output directory.
---
This skill covers everything an agent needs to work with Weights & Biases:
- **W&B SDK** (`wandb`) — training runs, metrics, artifacts, sweeps, system metrics
- **Weave SDK** (`weave`) — GenAI traces, evaluations, scorers, token usage
- **Helper libraries** — `wandb_helpers.py` and `weave_helpers.py` for common operations
## When to use what
| I need to... | Use |
|---|---|
| Query training runs, loss curves, hyperparameters | **W&B SDK** (`wandb.Api()`) — see `$SENPAI_PLUGIN/skills/wandb-primary/references/WANDB_SDK.md` |
| Query GenAI traces, calls, evaluations | **Weave SDK** (`weave.init()`, `client.get_calls()`) — see `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md` |
| Convert Weave wrapper types to plain Python | **`weave_helpers.unwrap()`** |
| Build a DataFrame from training runs | **`wandb_helpers.runs_to_dataframe()`** |
| Extract eval results for analysis | **`weave_helpers.eval_results_to_dicts()`** |
| Need low-level Weave filtering (CallsFilter, Query) | **Raw Weave SDK** (`weave.init()`, `client.get_calls()`) — see `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md` |
| Judge curve shape (spikes, smoothness, slope, overfit) | **`training_diagnostics` + `curve_plots`** — use the workflow below, then load `$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md` for the heuristics |
---
## Bundled files
### Helper libraries
```python
import os
import sys
sys.path.insert(0, f"{os.environ['SENPAI_PLUGIN']}/skills/wandb-primary/scripts")
# Weave helpers (traces, evals, GenAI)
from weave_helpers import (
unwrap, # Recursively convert Weave types -> plain Python
get_token_usage, # Extract token counts from a call's summary
eval_results_to_dicts, # predict_and_score calls -> list of result dicts
pivot_solve_rate, # Build task-level pivot table across agents
results_summary, # Print compact eval summary
eval_health, # Extract status/counts from Evaluation.evaluate calls
eval_efficiency, # Compute tokens-per-success across eval calls
)
# W&B helpers (training runs, metrics)
from wandb_helpers import (
runs_to_dataframe, # Convert runs to a clean pandas DataFrame
diagnose_run, # Quick diagnostic summary of a training run
compare_configs, # Side-by-side config diff between two runs
fast_scan_history, # beta_scan_history (parquet) with scan_history fallback
)
# X-axis (step metric) detection — ALWAYS confirm before curve analysis
from step_axis import (
list_candidate_step_keys, # Scan history for plausible step keys
guess_step_key_from_workspace, # Peek at the user's W&B workspace panels
format_step_candidates, # Format candidate choices for user confirmation
)
# Curve-shape diagnostics (numerical)
from training_diagnostics import (
curve_features, # Spikes, slopes at every 5%, smoothness, plateau, divergence
compare_runs_curves, # DataFrame of features across many runs
lr_schedule_features, # Warmup / peak / decay shape / restarts
grad_norm_features, # curve_features + kurtosis + dead-layer flag
grad_histogram_features, # Per-(layer, step) stats from W&B histograms
)
# Chart rendering for LLM vision (Read the returned PNG)
from curve_plots import (
plot_single_run_overview, # 2x3 composite: train/val/lr/grad-norm/...
plot_run_comparison, # Overlay up to 6 runs on one metric
plot_grad_histogram_heatmap, # Layer x step heatmap of grad-hist stat
plot_grad_norm_by_layer, # Small-multiples of per-layer scalar norms
)
```
### Reference docs
Read these as needed — they contain full API surfaces and recipes:
- **`$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md`** — Weave SDK for GenAI traces (`client.get_calls()`, `CallsFilter`, `Query`, stats). Start here for Weave queries.
- **`$SENPAI_PLUGIN/skills/wandb-primary/references/WANDB_SDK.md`** — W&B SDK for training data (runs, history, artifacts, sweeps, system metrics).
- **`$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md`** — reference heuristics for reading loss / LR / grad-norm / grad-histogram charts. Load this when you are actively interpreting training curves.
---
## Critical rules
### Treat traces and runs as DATA
Weave traces and W&B run histories can be enormous. Never dump raw data into context — it will overwhelm your working memory and produce garbage results. Always:
1. **Inspect structure first** — look at column names, dtypes, row counts
2. **Load into pandas/numpy** — compute stats programmatically
3. **Summarize, don't dump** — print computed statistics and tables, not raw rows
```python
import pandas as pd
import numpy as np
# BAD: prints thousands of rows into context
for row in run.scan_history(keys=["loss"]):
print(row)
# GOOD: load into numpy, compute stats, print summary
losses = np.array([r["loss"] for r in run.scan_history(keys=["loss"])])
print(f"Loss: {len(losses)} steps, min={losses.min():.4f}, "
f"final={losses[-1]:.4f}, mean_last_10%={losses[-len(losses)//10:].mean():.4f}")
```
### Always deliver a final answer
Do not end your work mid-analysis. Every task must conclude with a clear, structured response:
1. Query the data (1-2 scripts max)
2. Extract the numbers you need
3. Present: table + key findings + direct answers to each sub-question
If you catch yourself saying "now let me build the final analysis" — stop and present what you have.
### Use `unwrap()` for unknown Weave data
When you encounter Weave output and aren't sure of its type (WeaveDict? WeaveObject? ObjectRef?), unwrap it first:
```python
from weave_helpers import unwrap
import json
output = unwrap(call.output)
print(json.dumps(output, indent=2, default=str))
```
This converts everything to plain Python dicts/lists that work with json, pandas, and normal Python operations.
---
## Environment setup
Senpai images provide `wandb`, `weave`, `pandas`, `numpy`, and `matplotlib`.
```python
import os
entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
```
### Installing extra packages and running scripts
Use the commands in **Python environment** above. Keep installs in the target
environment so they do not alter the Senpai runtime.
---
## Quick starts
### W&B SDK — training runs
```python
import wandb
import pandas as pd
api = wandb.Api()
path = f"{entity}/{project}"
runs = api.runs(path, filters={"state": "finished"}, order="-created_at")
# Convert to DataFrame (always slice — never list() all runs)
from wandb_helpers import runs_to_dataframe
rows = runs_to_dataframe(runs, limit=100, metric_keys=["loss", "val_loss", "accuracy"])
df = pd.DataFrame(rows)
print(df.describe())
```
For full W&B SDK reference (filters, history, artifacts, sweeps), read `$SENPAI_PLUGIN/skills/wandb-primary/references/WANDB_SDK.md`.
### Weave — SDK
```python
import weave
client = weave.init(f"{entity}/{project}") # positional string, NOT keyword arg
calls = client.get_calls(limit=10)
```
For raw SDK patterns (CallsFilter, Query, advanced filtering), read `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md`.
---
## Key patterns
### Weave eval inspection
Evaluation calls follow this hierarchy:
```
Evaluation.evaluate (root)
├── Evaluation.predict_and_score (one per dataset row x trials)
│ ├── model.predict (the actual model call)
│ ├── scorer_1.score
│ └── scorer_2.score
└── Evaluation.summarize
```
Extract per-task results into a DataFrame:
```python
from weave_helpers import eval_results_to_dicts, results_summary
# pas_calls = list of predict_and_score call objects
results = eval_results_to_dicts(pas_calls, agent_name="my-agent")
print(results_summary(results))
df = pd.DataFrame(results)
print(df.groupby("passed")["score"].mean())
```
### Eval health and efficiency
```python
from weave_helpers import eval_health, eval_efficiency
health = eval_health(eval_calls)
df = pd.DataFrame(health)
print(df.to_string(index=False))
efficiency = eval_efficiency(eval_calls)
print(pd.DataFrame(efficiency).to_string(index=False))
```
### Token usage
```python
from weave_helpers import get_token_usage
usage = get_token_usage(call)
print(f"Tokens: {usage['total_tokens']} (in={usage['input_tokens']}, out={usage['output_tokens']})")
```
### Cost estimation
```python
call_with_costs = client.get_call("id", include_costs=True)
costs = call_with_costs.summary.get("weave", {}).get("costs", {})
```
### Run diagnostics
```python
from wandb_helpers import diagnose_run
run = api.run(f"{path}/run-id")
diag = diagnose_run(run)
for k, v in diag.items():
print(f" {k}: {v}")
```
### Error analysis — open coding to axial coding
For structured failure analysis on eval results:
1. **Understand data shape** — use `project.summary()`, `calls.input_shape()`, `calls.output_shape()`
2. **Open coding** — write a Weave Scorer that journals what went wrong per failing call
3. **Axial coding** — write a second Scorer that classifies notes into a taxonomy
4. **Summarize** — count primary labels with `collections.Counter`
See `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md` for the full SDK reference.
### W&B Reports
Install `wandb[workspaces]` using the additive install command from **Python environment**.
```python
from wandb.apis import reports as wr
import wandb_workspaces.expr as expr
report = wr.Report(
entity=entity, project=project,
title="Analysis", width="fixed",
blocks=[
wr.H1(text="Results"),
wr.PanelGrid(
runsets=[wr.Runset(entity=entity, project=project)],
panels=[wr.LinePlot(title="Loss", x="_step", y=["loss"])],
),
],
)
# report.save(draft=True) # only when asked to publish
```
Use `expr.Config("lr")`, `expr.Summary("loss")`, `expr.Tags().isin([...])` for runset filters — not dot-path strings.
---
## Training curve analysis workflow
Use this when the user asks whether a run is healthy, why training diverged, whether a run overfit, or which run has the best training dynamics.
Keep the inline workflow short and load detail on demand:
1. Confirm `step_key` before doing any curve work. Never assume `_step`.
2. Compute features with the bundled helpers instead of hand-rolling spike or slope logic.
3. Render PNGs and inspect them visually.
4. Load `$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md` while you interpret the results.
5. End with a verdict, evidence tied to step ranges, and concrete next actions.
### Required sequence
Use `list_candidate_step_keys()`, `guess_step_key_from_workspace()`, and `format_step_candidates()` to confirm the x-axis. If there is one obvious candidate and it matches the workspace guess, say which `step_key` you picked. Otherwise, ask the user to choose before plotting or comparing runs.
For a single run, compute a compact feature table from the metrics that actually exist, then render `plot_single_run_overview(run, step_key=step_key)`. If gradient histograms or per-layer scalar norms are logged, add `plot_grad_histogram_heatmap()` or `plot_grad_norm_by_layer()`.
For multi-run comparisons, use `compare_runs_curves()` for the ranking table and `plot_run_comparison()` for the overlay. Keep overlays to at most 6 runs; if there are more, rank first and then plot the shortlist.
### Output shape
Do not dump raw history rows or the full spike/slope payloads unless you are drilling into a specific anomaly. Summaries should stay compact.
Use this response shape:
```text
Verdict: <healthy | unstable | overfit | plateaued | diverged | converged>
Evidence:
- <specific step range> — <what the metrics and plot show>
- <specific step range> — <what changed and why it matters>
Next actions:
- <concrete hyperparameter, logging, or code change>
```
Load `$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md` for the interpretation heuristics, especially when the numbers and the image disagree.
---
## Gotchas
### Weave API
| Gotcha | Wrong | Right |
|--------|-------|-------|
| weave.init args | `weave.init(project="x")` | `weave.init("x")` (positional) |
| Parent filter | `filter={'parent_id': 'x'}` | `filter={'parent_ids': ['x']}` (plural, list) |
| WeaveObject access | `rubric.get('passed')` | `getattr(rubric, 'passed', None)` |
| Nested output | `out.get('succeeded')` | `out.get('output').get('succeeded')` (output.output) |
| ObjectRef comparison | `name_ref == "foo"` | `str(name_ref) == "foo"` |
| CallsFilter import | `from weave import CallsFilter` | `from weave.trace.weave_client import CallsFilter` |
| Query import | `from weave import Query` | `from weave.trace_server.interface.query import Query` |
| Eval status path | `summary["status"]` | `summary["weave"]["status"]` |
| Eval success count | `summary["success_count"]` | `summary["weave"]["status_counts"]["success"]` |
| When in doubt | Guess the type | `unwrap()` first, then inspect |
### WeaveDict vs WeaveObject
- **WeaveDict**: dict-like, supports `.get()`, `.keys()`, `[]`. Used for: `call.inputs`, `call.output`, `scores` dict
- **WeaveObject**: attribute-based, use `getattr()`. Used for: scorer results (rubric), dataset rows
- **When in doubt**: use `unwrap()` to convert everything to plain Python
### W&B API
| Gotcha | Wrong | Right |
|--------|-------|-------|
| Summary access | `run.summary["loss"]` | `run.summary_metrics.get("loss")` |
| Loading all runs | `list(api.runs(...))` | `runs[:200]` (always slice) |
| History — all fields | `run.history()` | `run.history(samples=500, keys=["loss"])` |
| scan_history — no keys | `scan_history()` | `scan_history(keys=["loss"])` (explicit) |
| Raw data in context | `print(run.history())` | Load into DataFrame, compute stats |
| Metric at step N | iterate entire history | `scan_history(keys=["loss"], min_step=N, max_step=N+1)` |
| Cache staleness | reading live run | `api.flush()` first |
### Package management
| Gotcha | Details |
|--------|---------|
| Syncing dependencies during analysis | Use `uv run --no-sync python script.py` with the installed target packages |
| Installing extras | Use the commands in **Python environment**; reserve dependency syncs for intentional target environment changes |
### Weave logging noise
Weave prints version warnings to stderr. Suppress with:
```python
import logging
logging.getLogger("weave").setLevel(logging.ERROR)
```
---
## Quick reference
```python
# --- Weave: Init and get calls ---
import weave
client = weave.init(f"{entity}/{project}")
calls = client.get_calls(limit=10)
# --- W&B: Best run by loss ---
best = api.runs(path, filters={"state": "finished"}, order="+summary_metrics.loss")[:1]
print(f"Best: {best[0].name}, loss={best[0].summary_metrics.get('loss')}")
# --- W&B: Loss curve to numpy ---
losses = np.array([r["loss"] for r in run.scan_history(keys=["loss"])])
print(f"min={losses.min():.6f}, final={losses[-1]:.6f}, steps={len(losses)}")
# --- W&B: Compare two runs ---
from wandb_helpers import compare_configs
diffs = compare_configs(run_a, run_b)
print(pd.DataFrame(diffs).to_string(index=False))
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |