Lambenthan/empiricalwiki / i18n/en/skills/exp-status
exp-status
VerifiedView the status of all running experiments; optionally auto-collect completed experiments and advance the pipeline
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.4 KB |
SKILL.md
View on GitHub---
description: View the status of all running experiments; optionally auto-collect completed experiments and advance the pipeline
argument-hint: "[--pipeline <slug>] [--collect-ready] [--auto-advance]"
---
# /exp-status
> Unified experiment status monitoring entry point.
> Scans all `running` experiments, performs a live status check on each (screen session / SSH),
> and outputs a status table (alive / anomaly / completed) to guide the user's next actions.
>
> When used with `/research --auto`, acts as a periodic checker scheduled by CronCreate:
> when all experiments in a pipeline are completed, automatically triggers `/research --start-from stage4`.
## Inputs
- No arguments (default): check all `running` experiments, print status table
- `--pipeline <slug>` (optional): check only experiments belonging to the specified pipeline; additionally print overall pipeline progress
- `--collect-ready` (optional): auto-call `/exp-run --collect` for all experiments whose session has already ended
- `--auto-advance` (optional, requires `--pipeline <slug>`): if all pipeline experiments are `completed`,
automatically trigger `/research --start-from stage4` without waiting for the user
## Outputs
- **Status report** (terminal output, all modes): list of experiments in running/anomaly/completed states
- `wiki/experiments/{slug}.md` — updated (outcome/key_result/status) when `--collect-ready` triggers Phase 4
- `wiki/outputs/pipeline-progress.md` — `--auto-advance` updates current_stage → stage4 (done internally by /research --start-from stage4)
- `wiki/log.md` — appended status check log
## Wiki Interaction
### Reads
- `wiki/experiments/*.md` — status, remote frontmatter (server/session/started), date_planned
- `wiki/outputs/pipeline-progress.md` — in `--pipeline` mode, identifies target experiments and monitoring_cron_id
### Writes
- `wiki/experiments/{slug}.md` — updated via /exp-run --collect in `--collect-ready` mode
- `wiki/outputs/pipeline-progress.md` — updated by /research when `--auto-advance` triggers Stage 4
- `wiki/log.md` — appended status check log
### Graph edges created
- None (result writes triggered indirectly via /exp-run --collect do not produce new edges)
## Workflow
**Precondition**: confirm working directory is the wiki project root (directory containing `wiki/`, `raw/`, `tools/`).
### Step 1: Collect Target Experiment List
1. **Determine check scope**:
- If `--pipeline <slug>` is specified:
- Read `wiki/outputs/pipeline-progress.md`, extract the slug list from the `stage3a_deployed` field
- If the file does not exist or slug does not match: report error, suggest running `/research` first or specifying manually
- Otherwise:
- Use Glob to scan `wiki/experiments/*.md`, filter for `status == running`
2. **If no running experiments**:
- Print a friendly message:
```
No running experiments found.
- To start an experiment: /exp-run <slug>
- To see all experiments: check wiki/experiments/
```
- Return
### Step 2: Check Status of Each Experiment
For each target experiment, execute in parallel (or sequentially):
1. **Read experiment page**: from `wiki/experiments/{slug}.md` get:
- `remote` block (if present, this is a remote experiment)
- `run_log` path
- `started` (from `remote.started` or `date_planned`, used to compute elapsed time)
- Deployment environment (has remote block → remote, otherwise → local)
2. **Check process status**:
- **Local**: `screen -ls | grep "exp-{slug}"`
- Has output → `alive: true`
- No output → `alive: false` (session is gone)
- **Remote**: `python3 tools/remote.py check --name "exp-{slug}"`
- Parse JSON: `alive`, `last_lines`, `anomalies`
3. **If alive == true**:
- Fetch recent logs (at most 20 lines):
- Local: `tail -20 {run_log}`
- Remote: use `last_lines` from the `check` command response
- Extract latest metric (loss, accuracy, step, etc. — grep the last metric line)
- Detect anomalies (NaN/OOM/Traceback/Inf): use `anomalies` field from `remote.py check` (remote), or manual grep (local)
- Compute elapsed time (current time − started)
- Classify as: `running` or `anomaly`
4. **If alive == false**:
- Classify as: `completed_pending_collect` (session gone but wiki status is still running)
- If wiki status is already `completed`: classify as `collected`
5. **Aggregate results**: build status dict `{slug: {state, elapsed, latest_metric, anomalies}}`
### Step 3: Print Status Report
```markdown
# Experiment Status — {YYYY-MM-DD HH:MM}
### 🔄 Running ({N})
| Experiment | Elapsed | Latest | Env |
|-----------|---------|--------|-----|
| [[exp-foo-baseline]] | 2.3h | loss: 0.42 | local |
| [[exp-foo-validation]] | 1.1h | step: 1200 | remote (gpu1) |
### ⚠️ Anomaly Detected ({N})
| Experiment | Elapsed | Issue | Action |
|-----------|---------|-------|--------|
| [[exp-foo-ablation]] | 0.8h | NaN loss at step 500 | Run `/exp-run exp-foo-ablation --collect` to inspect |
### ✅ Completed — Pending Collect ({N})
| Experiment | Finished (estimate) |
|-----------|---------------------|
| [[exp-foo-sanity]] | session gone |
### 📦 Already Collected ({N})
| Experiment | Outcome |
|-----------|---------|
| [[exp-foo-old]] | succeeded |
---
### Actions
```bash
# Collect all completed experiments at once:
/exp-status --collect-ready
# Collect a specific experiment:
/exp-run exp-foo-sanity --collect
# Pipeline progress (if in /research):
/exp-status --pipeline {pipeline-slug}
```
```
Append log:
```bash
python3 tools/research_wiki.py log wiki/ \
"exp-status | running: {N}, anomaly: {M}, pending-collect: {K}"
```
### Step 4: --collect-ready Auto-Collect (if specified)
For each `completed_pending_collect` experiment, call `/exp-run --collect`:
```
Skill: exp-run
Args: "{slug} --collect"
```
Collect each completed experiment sequentially (not in parallel, to avoid concurrent wiki writes).
After all collections are done, re-print the updated status report.
### Step 5: --auto-advance Pipeline Advance (if both --pipeline and --auto-advance are specified)
1. **Check pipeline completion condition**:
- Read `stage3a_deployed` list from `wiki/outputs/pipeline-progress.md`
- Check the status of each slug's `wiki/experiments/{slug}.md`
- **Condition met**: all experiments have status == `completed`
2. **If condition is not met** (some experiments still running or pending collect):
- Print current progress: `Pipeline {slug}: {M}/{N} experiments completed`
- Return (do not advance)
- Cron will trigger again in 30 minutes
3. **If condition is met (all experiments completed)**:
a. **Print notification and trigger Stage 4**:
- Print:
```
✅ All experiments completed for pipeline {slug}!
Advancing to Stage 4 (Verdict & Iteration)...
```
- Append log:
```bash
python3 tools/research_wiki.py log wiki/ \
"exp-status | pipeline {slug}: all experiments done, advancing to stage4"
```
- Trigger next stage:
```
Skill: research
Args: "--start-from stage4"
```
## Constraints
- **Read-only in non --collect-ready mode**: without `--collect-ready`, do not modify any wiki files
- **`--auto-advance` requires `--pipeline`**: using `--auto-advance` alone is invalid, report an error
- **Status checks must be non-blocking**: each experiment check should complete quickly (single SSH check or screen -ls)
- **Anomalies are not auto-fixed**: `/exp-status` only reports anomalies; fixes require the user to manually call `/exp-run --collect`
- **pipeline-progress.md must exist**: in `--pipeline` mode, if the file is missing, report an error
## Error Handling
- **No running experiments**: print friendly message, not an error; provide next step suggestions
- **`--pipeline` but pipeline-progress.md does not exist**: report error "Pipeline progress file not found. Run `/research <direction>` first or check wiki/outputs/"
- **`--auto-advance` without `--pipeline`**: report error "--auto-advance requires --pipeline <slug>"
- **SSH connection fails** (remote experiment): mark that experiment as `check_failed`, note it in the report, continue checking other experiments
- **screen -ls returns nothing**: does not mean the experiment failed — may be a brief delay; mark as `completed_pending_collect`
- **`/exp-run --collect` fails** (`--collect-ready` mode): record the failure, continue collecting other experiments, report all failures at the end
## Dependencies
### Skills(via Skill tool)
- `/exp-run` — call collect phase in `--collect-ready` mode
- `/research` — trigger Stage 4 via `--auto-advance`
### Tools(via Bash)
- `python3 tools/remote.py check --name "exp-{slug}"` — remote experiment status check
- `python3 tools/remote.py tail-log --name "exp-{slug}" --lines 20` — fetch remote logs
- `python3 tools/research_wiki.py set-meta <path> <field> <value>` — update pipeline-progress
- `python3 tools/research_wiki.py log wiki/ "<message>"` — append log
- `screen -ls` — local process status
- `tail -20 {log}` — fetch local logs
### Claude Code Native
- `Read` — read experiment pages and pipeline-progress
- `Write` — update pipeline-progress status
- `Glob` — scan wiki/experiments/*.md
- `Bash` — screen/tail and other system commands
- `Skill` — call /exp-run --collect and /research
### Called by
- CronCreate schedule (created by `/research --auto` Stage 3b: triggers every 30 minutes)
- User directly
- `/research` Stage 3b (in interactive mode, suggested to user)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |