Skip to content

Lambenthan/empiricalwiki / i18n/en/skills/exp-eval

exp-eval

Verified

Experiment verdict gate — Review LLM independently judges results → 4 verdict paths → auto-update claims confidence, ideas status, graph edges

/exp-eval

Convert completed experiment results into wiki knowledge updates. Review LLM acts as an impartial judge (following cross-model-review), independently evaluating how experimental results affect the target claim. Four verdict paths: supported → claim↑ + idea validated / partially_supported → supplementary experiments / not_supported → claim↓ + idea failed / inconclusive → debug. Auto-updates claims confidence and evidence, ideas status, and graph edges.

Inputs

  • experiment: slug from wiki/experiments/ (status must be completed)
  • --auto (optional): automatic mode — do not pause for user confirmation before wiki updates (used when called by /research)

Outputs

  • wiki/claims/{slug}.md — updated confidence, status, evidence list
  • wiki/ideas/{slug}.md — updated status (validated/failed), pilot_result, failure_reason
  • wiki/experiments/{slug}.md — ## Claim updates section filled in
  • wiki/graph/edges.jsonl — new supports/invalidates edges added
  • wiki/graph/context_brief.md — rebuilt
  • wiki/graph/open_questions.md — rebuilt
  • wiki/log.md — appended log entry
  • VERDICT_REPORT (printed to terminal) — verdict result, wiki change summary, next step suggestions

Wiki Interaction

Reads

  • wiki/experiments/{slug}.md — experiment results: outcome, key_result, metrics, full Results section
  • wiki/claims/{target-claim}.md — target claim current state: status, confidence, evidence list
  • wiki/ideas/{linked-idea}.md — linked idea current state
  • wiki/experiments/*.md — other experiments on the same claim (aggregate assessment)
  • wiki/graph/context_brief.md — global context
  • .claude/skills/shared-references/cross-model-review.md — reviewer independence principle

Writes

  • wiki/claims/{target-claim}.md — update status, confidence, evidence, date_updated
  • wiki/ideas/{linked-idea}.md — update status, pilot_result, failure_reason, date_resolved
  • wiki/experiments/{slug}.md — fill in ## Claim updates section
  • wiki/graph/edges.jsonl — add supports or invalidates edges
  • wiki/graph/context_brief.md — rebuild
  • wiki/graph/open_questions.md — rebuild
  • wiki/log.md — append operation log

Graph edges created

  • supports: experiment → claim (experiment supports the claim) — verdict = supported or partially_supported
  • invalidates: experiment → claim (experiment refutes the claim) — verdict = not_supported

Workflow

Precondition:

  1. Confirm working directory is the wiki project root (directory containing wiki/, raw/, tools/)
  2. Confirm experiment status == completed (incomplete experiments cannot be evaluated)

Step 1: Load Context

  1. Read experiment page wiki/experiments/{slug}.md:

    • outcome (succeeded/failed/inconclusive)
    • key_result
    • target_claim slug
    • linked_idea slug
    • metrics and full Results section
    • hypothesis
  2. Read target claim wiki/claims/{target-claim}.md:

    • Current status and confidence
    • Existing evidence list
    • Conditions and scope
  3. Read linked idea wiki/ideas/{linked-idea}.md (if it exists):

    • Current status
    • Hypothesis
  4. Load other experiments on the same claim:

    • Glob: wiki/experiments/*.md, filter target_claim == same claim
    • Summarize existing experiment results (for aggregate claim confidence assessment)
  5. Read global context:

    • wiki/graph/context_brief.md
  6. Read cross-model-review.md: confirm Review LLM independence principle

Step 2: Review LLM Verdict (Cross-Model Verdict)

Follow cross-model-review.md: do not send Claude's pre-judgment to Review LLM.

mcp__llm-review__chat:
  system: "You are an impartial scientific judge evaluating whether experimental
           results support or refute a research claim. Be rigorous and objective.
           Consider: statistical significance, effect size, experimental validity,
           potential confounds, and whether the results generalize beyond the
           specific setup tested."
  message: |
    ## Claim Under Test
    Title: {claim title}
    Statement: {claim statement from ## Statement section}
    Current status: {status}
    Current confidence: {confidence}
    Conditions: {conditions and scope}

    ## Experiment
    Title: {experiment title}
    Hypothesis: {hypothesis}
    Setup: {model, dataset, hardware, framework}
    Metrics: {metrics list}

    ## Results
    {full Results section from experiment page}

    ## Key Finding
    {key_result}

    ## Other Experiments on This Claim
    {summary of other experiments' outcomes on the same claim, if any}

    ## Your Task
    Provide your verdict:
    1. **Verdict**: One of: supported / partially_supported / not_supported / inconclusive
    2. **Confidence adjustment**: Suggest new confidence value (0.0-1.0) with reasoning
    3. **Evidence strength**: weak / moderate / strong
    4. **Key reasoning**: 2-3 sentences explaining your verdict
    5. **Concerns**: Any methodological concerns or limitations
    6. **Suggested next steps**: What would strengthen or clarify this result?

Record Review LLM's verdict.

Step 3: Claude Synthesis

  1. Form Claude's independent verdict (after reading Review LLM's verdict, Claude also analyzes independently):

    • Based on experimental results, claim context, and aggregate evidence from other experiments
    • Form Claude's own verdict and confidence suggestion
  2. Synthesize both verdicts (follow cross-model-review.md composing rules):

    • Both agree (same verdict): use that verdict, average the confidence, high certainty
    • Both disagree:
      • Explicitly flag the disagreement
      • Take the more conservative verdict (supported > partially_supported > not_supported)
      • Use the lower confidence value
      • Detail the disagreement reason in the report
    • Fatal findings take priority: if either party finds a methodological issue (data leakage, unfair comparison), that finding takes precedence
  3. Determine final verdict: verdict + new_confidence + evidence_strength

Step 4: Update Wiki Based on Verdict

If --auto is not set: display verdict and planned changes first, wait for user confirmation.

Path A: SUPPORTED (experiment supports claim)

  1. Update claim:

    • confidence: ↑ adjust to new value (typically +0.1~0.3)
    • status: adjust based on new confidence
      • confidence >= 0.7 → supported
      • confidence 0.4–0.7 → weakly_supported
    • evidence: append new entry {source: experiment-slug, type: supports, strength: strong/moderate, detail: key_result}
    • date_updated: today's date
  2. Update idea (if it exists and status is in_progress/tested):

    • If all linked claims are supported/weakly_supported:
      • status: validated
      • pilot_result: key_result summary
      • date_resolved: today's date
  3. Add graph edge:

    python3 tools/research_wiki.py add-edge wiki/ \
      --from "experiments/{slug}" --to "claims/{target-claim}" \
      --type supports --evidence "{key_result}"
    
  4. Suggest next steps: /paper-plan or continue ablation/robustness experiments

Path B: PARTIALLY_SUPPORTED (partial support)

  1. Update claim:

    • confidence: minor adjustment (+0.05~0.15)
    • evidence: append {type: supports, strength: weak, detail: ...}
    • date_updated: today's date
  2. Add graph edge:

    python3 tools/research_wiki.py add-edge wiki/ \
      --from "experiments/{slug}" --to "claims/{target-claim}" \
      --type supports --evidence "Partially supported: {limitation}"
    
  3. Suggest supplementary experiments:

    • Specify what evidence is missing
    • Suggest using /exp-design to design supplementary experiments
    • If Review LLM-flagged concerns are addressable by experiment, suggest concrete experiment direction
  4. Idea status unchanged: keep in_progress, wait for more evidence

Path C: NOT_SUPPORTED (experiment does not support claim)

  1. Update claim:

    • confidence: ↓ significantly lower (typically -0.2~0.4)
    • status: if confidence < 0.3 → challenged
    • evidence: append {type: invalidates, strength: strong/moderate, detail: ...}
    • date_updated: today's date
  2. Update idea (if it exists):

    • status: failed
    • failure_reason: specific reason for failure (extracted from experiment results and Review LLM analysis)
    • date_resolved: today's date
    • Note: failure_reason is anti-repetition memory — must be written clearly, explaining why it failed
  3. Add graph edge:

    python3 tools/research_wiki.py add-edge wiki/ \
      --from "experiments/{slug}" --to "claims/{target-claim}" \
      --type invalidates --evidence "{failure_reason}"
    
  4. Suggest next steps:

    • Analyze the failure reason
    • Consider pivoting (new idea addressing the same gap while avoiding the known failure)
    • Suggest /ideate to generate alternatives

Path D: INCONCLUSIVE (results are uncertain)

  1. Do not modify claim status/confidence: insufficient evidence to make a judgment

  2. Update experiment page: outcome is already inconclusive (set by /exp-run)

  3. Suggest debugging:

    • Data issue? Implementation bug? Wrong metric?
    • Too much variance? More seeds needed?
    • Experiment setup not aligned with claim?
  4. Idea status unchanged: keep current status

All Paths (common steps)

  1. Fill in ## Claim updates section of the experiment page:

    ## Claim updates
    - **Verdict**: {supported/partially_supported/not_supported/inconclusive}
    - **Claim**: [[{target-claim}]] confidence {old} → {new}
    - **Judge agreement**: {Claude and Review LLM agreed / disagreed on ...}
    - **Date**: YYYY-MM-DD
    
  2. Update index.md (if claim status changed)

  3. Rebuild derived data:

    python3 tools/research_wiki.py rebuild-context-brief wiki/
    python3 tools/research_wiki.py rebuild-open-questions wiki/
    
  4. Append log:

    python3 tools/research_wiki.py log wiki/ \
      "exp-eval | {slug} → {target-claim} | verdict: {verdict} | confidence: {old}→{new}"
    
  5. Print VERDICT_REPORT to terminal:

    # Verdict Report: {experiment title}
    
    ## Verdict: {SUPPORTED / PARTIALLY_SUPPORTED / NOT_SUPPORTED / INCONCLUSIVE}
    
    ## Judge Assessment
    | | Claude | Review LLM | Final |
    |---|-------|------|-------|
    | Verdict | {verdict} | {verdict} | {verdict} |
    | Confidence | {value} | {value} | {value} |
    | Evidence strength | {strength} | {strength} | {strength} |
    
    ## Key Reasoning
    {2-3 sentences from Review LLM + Claude synthesis}
    
    ## Wiki Changes
    | Entity | Field | Before | After |
    |--------|-------|--------|-------|
    | claims/{slug} | confidence | {old} | {new} |
    | claims/{slug} | status | {old} | {new} |
    | ideas/{slug} | status | {old} | {new} |
    
    ## Graph Edges Added
    - experiments/{slug} → claims/{target} (supports/invalidates)
    
    ## Concerns
    {methodological concerns from Review LLM}
    
    ## Next Steps
    - {path-specific suggestions}
    
    ## Wiki Growth
    | Metric | Before | After | Delta |
    |--------|--------|-------|-------|
    | Claims updated | — | — | {N} |
    | Edges | {before} | {after} | +{delta} |
    | Maturity | {level} | {level} | {unchanged/upgraded} |
    (Data from comparing `python3 tools/research_wiki.py maturity wiki/ --json` calls at the start of Step 1 and end of Step 4.)
    

Constraints

  • Only process completed experiments: experiments with status != completed are refused; prompt user to use /exp-run first
  • Reviewer independence: strictly follow cross-model-review.md — do not send Claude's pre-judgment to Review LLM
  • Confidence range 0.0–1.0: updated confidence must not exceed this range
  • failure_reason must be specific: the not_supported path's failure_reason cannot be vague (e.g. "experiment failed") — must state the concrete reason
  • Do not delete claims: even when not_supported, only challenge or lower confidence; do not delete the claim page. In extreme cases (multiple consistent refutations, confidence → 0), set status to deprecated rather than deleting
  • Graph edges via tools/research_wiki.py: do not manually edit edges.jsonl
  • Conservative principle: when Claude and Review LLM verdicts disagree, use the more conservative verdict
  • Idea status advances only forward: proposed → in_progress → tested → validated/failed, irreversible
  • Assess claim using all experiments: consider not just the current experiment but also other experiments on the same claim

Error Handling

  • Experiment not found: prompt user to check slug, list candidates in wiki/experiments/ with status=completed
  • Experiment not completed: report status, suggest running /exp-run {slug} or /exp-run {slug} --check
  • Target claim does not exist: create new claim page (status: proposed, confidence: 0.3), note "auto-created by exp-eval"
  • Linked idea does not exist: skip idea update, only update claim, note in report
  • Review LLM unavailable: fall back to Claude single-model verdict, note "single-model verdict, cross-model verification unavailable" in report, suggest user confirm later
  • Claim was modified by another experiment: read the latest state, make adjustments based on current confidence (do not overwrite other experiments' contributions)
  • Results data missing: if the experiment page's Results section is empty, prompt user to run /exp-run {slug} --check first

Dependencies

Tools(via Bash)

  • python3 tools/research_wiki.py add-edge wiki/ ... — add graph edge
  • python3 tools/research_wiki.py rebuild-context-brief wiki/ — rebuild query_pack
  • python3 tools/research_wiki.py rebuild-open-questions wiki/ — rebuild gap_map
  • python3 tools/research_wiki.py log wiki/ "<message>" — append log

MCP Servers

  • mcp__llm-review__chat — Step 2 Review LLM independent verdict

Claude Code Native

  • Read — read wiki pages
  • Glob — find other experiments on the same claim
  • Edit — update wiki pages

Shared References

  • .claude/skills/shared-references/cross-model-review.md — Review LLM independence principle (required reading)

Called by

  • /research Stage 4 (verdict and iteration stage)
  • User directly
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall