Skip to content

Lambenthan/empiricalwiki / i18n/en/skills/exp-status

exp-status

Verified

View the status of all running experiments; optionally auto-collect completed experiments and advance the pipeline

/exp-status

Unified experiment status monitoring entry point. Scans all running experiments, performs a live status check on each (screen session / SSH), and outputs a status table (alive / anomaly / completed) to guide the user's next actions.

When used with /research --auto, acts as a periodic checker scheduled by CronCreate: when all experiments in a pipeline are completed, automatically triggers /research --start-from stage4.

Inputs

  • No arguments (default): check all running experiments, print status table
  • --pipeline <slug> (optional): check only experiments belonging to the specified pipeline; additionally print overall pipeline progress
  • --collect-ready (optional): auto-call /exp-run --collect for all experiments whose session has already ended
  • --auto-advance (optional, requires --pipeline <slug>): if all pipeline experiments are completed, automatically trigger /research --start-from stage4 without waiting for the user

Outputs

  • Status report (terminal output, all modes): list of experiments in running/anomaly/completed states
  • wiki/experiments/{slug}.md — updated (outcome/key_result/status) when --collect-ready triggers Phase 4
  • wiki/outputs/pipeline-progress.md — --auto-advance updates current_stage → stage4 (done internally by /research --start-from stage4)
  • wiki/log.md — appended status check log

Wiki Interaction

Reads

  • wiki/experiments/*.md — status, remote frontmatter (server/session/started), date_planned
  • wiki/outputs/pipeline-progress.md — in --pipeline mode, identifies target experiments and monitoring_cron_id

Writes

  • wiki/experiments/{slug}.md — updated via /exp-run --collect in --collect-ready mode
  • wiki/outputs/pipeline-progress.md — updated by /research when --auto-advance triggers Stage 4
  • wiki/log.md — appended status check log

Graph edges created

  • None (result writes triggered indirectly via /exp-run --collect do not produce new edges)

Workflow

Precondition: confirm working directory is the wiki project root (directory containing wiki/, raw/, tools/).

Step 1: Collect Target Experiment List

  1. Determine check scope:

    • If --pipeline <slug> is specified:
      • Read wiki/outputs/pipeline-progress.md, extract the slug list from the stage3a_deployed field
      • If the file does not exist or slug does not match: report error, suggest running /research first or specifying manually
    • Otherwise:
      • Use Glob to scan wiki/experiments/*.md, filter for status == running
  2. If no running experiments:

    • Print a friendly message:
      No running experiments found.
      - To start an experiment: /exp-run <slug>
      - To see all experiments: check wiki/experiments/
      
    • Return

Step 2: Check Status of Each Experiment

For each target experiment, execute in parallel (or sequentially):

  1. Read experiment page: from wiki/experiments/{slug}.md get:

    • remote block (if present, this is a remote experiment)
    • run_log path
    • started (from remote.started or date_planned, used to compute elapsed time)
    • Deployment environment (has remote block → remote, otherwise → local)
  2. Check process status:

    • Local: screen -ls | grep "exp-{slug}"
      • Has output → alive: true
      • No output → alive: false (session is gone)
    • Remote: python3 tools/remote.py check --name "exp-{slug}"
      • Parse JSON: alive, last_lines, anomalies
  3. If alive == true:

    • Fetch recent logs (at most 20 lines):
      • Local: tail -20 {run_log}
      • Remote: use last_lines from the check command response
    • Extract latest metric (loss, accuracy, step, etc. — grep the last metric line)
    • Detect anomalies (NaN/OOM/Traceback/Inf): use anomalies field from remote.py check (remote), or manual grep (local)
    • Compute elapsed time (current time − started)
    • Classify as: running or anomaly
  4. If alive == false:

    • Classify as: completed_pending_collect (session gone but wiki status is still running)
    • If wiki status is already completed: classify as collected
  5. Aggregate results: build status dict {slug: {state, elapsed, latest_metric, anomalies}}

Step 3: Print Status Report

# Experiment Status — {YYYY-MM-DD HH:MM}

### 🔄 Running ({N})
| Experiment | Elapsed | Latest | Env |
|-----------|---------|--------|-----|
| [[exp-foo-baseline]] | 2.3h | loss: 0.42 | local |
| [[exp-foo-validation]] | 1.1h | step: 1200 | remote (gpu1) |

### ⚠️ Anomaly Detected ({N})
| Experiment | Elapsed | Issue | Action |
|-----------|---------|-------|--------|
| [[exp-foo-ablation]] | 0.8h | NaN loss at step 500 | Run `/exp-run exp-foo-ablation --collect` to inspect |

### ✅ Completed — Pending Collect ({N})
| Experiment | Finished (estimate) |
|-----------|---------------------|
| [[exp-foo-sanity]] | session gone |

### 📦 Already Collected ({N})
| Experiment | Outcome |
|-----------|---------|
| [[exp-foo-old]] | succeeded |

---
### Actions
```bash
# Collect all completed experiments at once:
/exp-status --collect-ready

# Collect a specific experiment:
/exp-run exp-foo-sanity --collect

# Pipeline progress (if in /research):
/exp-status --pipeline {pipeline-slug}

Append log:
```bash
python3 tools/research_wiki.py log wiki/ \
  "exp-status | running: {N}, anomaly: {M}, pending-collect: {K}"

Step 4: --collect-ready Auto-Collect (if specified)

For each completed_pending_collect experiment, call /exp-run --collect:

Skill: exp-run
Args: "{slug} --collect"

Collect each completed experiment sequentially (not in parallel, to avoid concurrent wiki writes).

After all collections are done, re-print the updated status report.

Step 5: --auto-advance Pipeline Advance (if both --pipeline and --auto-advance are specified)

  1. Check pipeline completion condition:

    • Read stage3a_deployed list from wiki/outputs/pipeline-progress.md
    • Check the status of each slug's wiki/experiments/{slug}.md
    • Condition met: all experiments have status == completed
  2. If condition is not met (some experiments still running or pending collect):

    • Print current progress: Pipeline {slug}: {M}/{N} experiments completed
    • Return (do not advance)
    • Cron will trigger again in 30 minutes
  3. If condition is met (all experiments completed):

    a. Print notification and trigger Stage 4:

    • Print:
      ✅ All experiments completed for pipeline {slug}!
      Advancing to Stage 4 (Verdict & Iteration)...
      
    • Append log:
      python3 tools/research_wiki.py log wiki/ \
        "exp-status | pipeline {slug}: all experiments done, advancing to stage4"
      
    • Trigger next stage:
      Skill: research
      Args: "--start-from stage4"
      

Constraints

  • Read-only in non --collect-ready mode: without --collect-ready, do not modify any wiki files
  • --auto-advance requires --pipeline: using --auto-advance alone is invalid, report an error
  • Status checks must be non-blocking: each experiment check should complete quickly (single SSH check or screen -ls)
  • Anomalies are not auto-fixed: /exp-status only reports anomalies; fixes require the user to manually call /exp-run --collect
  • pipeline-progress.md must exist: in --pipeline mode, if the file is missing, report an error

Error Handling

  • No running experiments: print friendly message, not an error; provide next step suggestions
  • --pipeline but pipeline-progress.md does not exist: report error "Pipeline progress file not found. Run /research <direction> first or check wiki/outputs/"
  • --auto-advance without --pipeline: report error "--auto-advance requires --pipeline "
  • SSH connection fails (remote experiment): mark that experiment as check_failed, note it in the report, continue checking other experiments
  • screen -ls returns nothing: does not mean the experiment failed — may be a brief delay; mark as completed_pending_collect
  • /exp-run --collect fails (--collect-ready mode): record the failure, continue collecting other experiments, report all failures at the end

Dependencies

Skills(via Skill tool)

  • /exp-run — call collect phase in --collect-ready mode
  • /research — trigger Stage 4 via --auto-advance

Tools(via Bash)

  • python3 tools/remote.py check --name "exp-{slug}" — remote experiment status check
  • python3 tools/remote.py tail-log --name "exp-{slug}" --lines 20 — fetch remote logs
  • python3 tools/research_wiki.py set-meta <path> <field> <value> — update pipeline-progress
  • python3 tools/research_wiki.py log wiki/ "<message>" — append log
  • screen -ls — local process status
  • tail -20 {log} — fetch local logs

Claude Code Native

  • Read — read experiment pages and pipeline-progress
  • Write — update pipeline-progress status
  • Glob — scan wiki/experiments/*.md
  • Bash — screen/tail and other system commands
  • Skill — call /exp-run --collect and /research

Called by

  • CronCreate schedule (created by /research --auto Stage 3b: triggers every 30 minutes)
  • User directly
  • /research Stage 3b (in interactive mode, suggested to user)
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall