Lambenthan/empiricalwiki / i18n/en/skills/exp-eval
exp-eval
VerifiedExperiment verdict gate — Review LLM independently judges results → 4 verdict paths → auto-update claims confidence, ideas status, graph edges
/exp-eval
Convert completed experiment results into wiki knowledge updates. Review LLM acts as an impartial judge (following cross-model-review), independently evaluating how experimental results affect the target claim. Four verdict paths: supported → claim↑ + idea validated / partially_supported → supplementary experiments / not_supported → claim↓ + idea failed / inconclusive → debug. Auto-updates claims confidence and evidence, ideas status, and graph edges.
Inputs
experiment: slug from wiki/experiments/ (status must becompleted)--auto(optional): automatic mode — do not pause for user confirmation before wiki updates (used when called by /research)
Outputs
wiki/claims/{slug}.md— updated confidence, status, evidence listwiki/ideas/{slug}.md— updated status (validated/failed), pilot_result, failure_reasonwiki/experiments/{slug}.md—## Claim updatessection filled inwiki/graph/edges.jsonl— new supports/invalidates edges addedwiki/graph/context_brief.md— rebuiltwiki/graph/open_questions.md— rebuiltwiki/log.md— appended log entry- VERDICT_REPORT (printed to terminal) — verdict result, wiki change summary, next step suggestions
Wiki Interaction
Reads
wiki/experiments/{slug}.md— experiment results: outcome, key_result, metrics, full Results sectionwiki/claims/{target-claim}.md— target claim current state: status, confidence, evidence listwiki/ideas/{linked-idea}.md— linked idea current statewiki/experiments/*.md— other experiments on the same claim (aggregate assessment)wiki/graph/context_brief.md— global context.claude/skills/shared-references/cross-model-review.md— reviewer independence principle
Writes
wiki/claims/{target-claim}.md— update status, confidence, evidence, date_updatedwiki/ideas/{linked-idea}.md— update status, pilot_result, failure_reason, date_resolvedwiki/experiments/{slug}.md— fill in## Claim updatessectionwiki/graph/edges.jsonl— add supports or invalidates edgeswiki/graph/context_brief.md— rebuildwiki/graph/open_questions.md— rebuildwiki/log.md— append operation log
Graph edges created
supports: experiment → claim (experiment supports the claim) — verdict = supported or partially_supportedinvalidates: experiment → claim (experiment refutes the claim) — verdict = not_supported
Workflow
Precondition:
- Confirm working directory is the wiki project root (directory containing
wiki/,raw/,tools/) - Confirm experiment status ==
completed(incomplete experiments cannot be evaluated)
Step 1: Load Context
-
Read experiment page
wiki/experiments/{slug}.md:- outcome (succeeded/failed/inconclusive)
- key_result
- target_claim slug
- linked_idea slug
- metrics and full Results section
- hypothesis
-
Read target claim
wiki/claims/{target-claim}.md:- Current status and confidence
- Existing evidence list
- Conditions and scope
-
Read linked idea
wiki/ideas/{linked-idea}.md(if it exists):- Current status
- Hypothesis
-
Load other experiments on the same claim:
- Glob:
wiki/experiments/*.md, filter target_claim == same claim - Summarize existing experiment results (for aggregate claim confidence assessment)
- Glob:
-
Read global context:
wiki/graph/context_brief.md
-
Read cross-model-review.md: confirm Review LLM independence principle
Step 2: Review LLM Verdict (Cross-Model Verdict)
Follow cross-model-review.md: do not send Claude's pre-judgment to Review LLM.
mcp__llm-review__chat:
system: "You are an impartial scientific judge evaluating whether experimental
results support or refute a research claim. Be rigorous and objective.
Consider: statistical significance, effect size, experimental validity,
potential confounds, and whether the results generalize beyond the
specific setup tested."
message: |
## Claim Under Test
Title: {claim title}
Statement: {claim statement from ## Statement section}
Current status: {status}
Current confidence: {confidence}
Conditions: {conditions and scope}
## Experiment
Title: {experiment title}
Hypothesis: {hypothesis}
Setup: {model, dataset, hardware, framework}
Metrics: {metrics list}
## Results
{full Results section from experiment page}
## Key Finding
{key_result}
## Other Experiments on This Claim
{summary of other experiments' outcomes on the same claim, if any}
## Your Task
Provide your verdict:
1. **Verdict**: One of: supported / partially_supported / not_supported / inconclusive
2. **Confidence adjustment**: Suggest new confidence value (0.0-1.0) with reasoning
3. **Evidence strength**: weak / moderate / strong
4. **Key reasoning**: 2-3 sentences explaining your verdict
5. **Concerns**: Any methodological concerns or limitations
6. **Suggested next steps**: What would strengthen or clarify this result?
Record Review LLM's verdict.
Step 3: Claude Synthesis
-
Form Claude's independent verdict (after reading Review LLM's verdict, Claude also analyzes independently):
- Based on experimental results, claim context, and aggregate evidence from other experiments
- Form Claude's own verdict and confidence suggestion
-
Synthesize both verdicts (follow cross-model-review.md composing rules):
- Both agree (same verdict): use that verdict, average the confidence, high certainty
- Both disagree:
- Explicitly flag the disagreement
- Take the more conservative verdict (supported > partially_supported > not_supported)
- Use the lower confidence value
- Detail the disagreement reason in the report
- Fatal findings take priority: if either party finds a methodological issue (data leakage, unfair comparison), that finding takes precedence
-
Determine final verdict: verdict + new_confidence + evidence_strength
Step 4: Update Wiki Based on Verdict
If --auto is not set: display verdict and planned changes first, wait for user confirmation.
Path A: SUPPORTED (experiment supports claim)
-
Update claim:
- confidence: ↑ adjust to new value (typically +0.1~0.3)
- status: adjust based on new confidence
- confidence >= 0.7 →
supported - confidence 0.4–0.7 →
weakly_supported
- confidence >= 0.7 →
- evidence: append new entry
{source: experiment-slug, type: supports, strength: strong/moderate, detail: key_result} - date_updated: today's date
-
Update idea (if it exists and status is in_progress/tested):
- If all linked claims are supported/weakly_supported:
- status:
validated - pilot_result: key_result summary
- date_resolved: today's date
- status:
- If all linked claims are supported/weakly_supported:
-
Add graph edge:
python3 tools/research_wiki.py add-edge wiki/ \ --from "experiments/{slug}" --to "claims/{target-claim}" \ --type supports --evidence "{key_result}" -
Suggest next steps:
/paper-planor continue ablation/robustness experiments
Path B: PARTIALLY_SUPPORTED (partial support)
-
Update claim:
- confidence: minor adjustment (+0.05~0.15)
- evidence: append
{type: supports, strength: weak, detail: ...} - date_updated: today's date
-
Add graph edge:
python3 tools/research_wiki.py add-edge wiki/ \ --from "experiments/{slug}" --to "claims/{target-claim}" \ --type supports --evidence "Partially supported: {limitation}" -
Suggest supplementary experiments:
- Specify what evidence is missing
- Suggest using
/exp-designto design supplementary experiments - If Review LLM-flagged concerns are addressable by experiment, suggest concrete experiment direction
-
Idea status unchanged: keep in_progress, wait for more evidence
Path C: NOT_SUPPORTED (experiment does not support claim)
-
Update claim:
- confidence: ↓ significantly lower (typically -0.2~0.4)
- status: if confidence < 0.3 →
challenged - evidence: append
{type: invalidates, strength: strong/moderate, detail: ...} - date_updated: today's date
-
Update idea (if it exists):
- status:
failed - failure_reason: specific reason for failure (extracted from experiment results and Review LLM analysis)
- date_resolved: today's date
- Note: failure_reason is anti-repetition memory — must be written clearly, explaining why it failed
- status:
-
Add graph edge:
python3 tools/research_wiki.py add-edge wiki/ \ --from "experiments/{slug}" --to "claims/{target-claim}" \ --type invalidates --evidence "{failure_reason}" -
Suggest next steps:
- Analyze the failure reason
- Consider pivoting (new idea addressing the same gap while avoiding the known failure)
- Suggest
/ideateto generate alternatives
Path D: INCONCLUSIVE (results are uncertain)
-
Do not modify claim status/confidence: insufficient evidence to make a judgment
-
Update experiment page: outcome is already inconclusive (set by /exp-run)
-
Suggest debugging:
- Data issue? Implementation bug? Wrong metric?
- Too much variance? More seeds needed?
- Experiment setup not aligned with claim?
-
Idea status unchanged: keep current status
All Paths (common steps)
-
Fill in
## Claim updatessection of the experiment page:## Claim updates - **Verdict**: {supported/partially_supported/not_supported/inconclusive} - **Claim**: [[{target-claim}]] confidence {old} → {new} - **Judge agreement**: {Claude and Review LLM agreed / disagreed on ...} - **Date**: YYYY-MM-DD -
Update index.md (if claim status changed)
-
Rebuild derived data:
python3 tools/research_wiki.py rebuild-context-brief wiki/ python3 tools/research_wiki.py rebuild-open-questions wiki/ -
Append log:
python3 tools/research_wiki.py log wiki/ \ "exp-eval | {slug} → {target-claim} | verdict: {verdict} | confidence: {old}→{new}" -
Print VERDICT_REPORT to terminal:
# Verdict Report: {experiment title} ## Verdict: {SUPPORTED / PARTIALLY_SUPPORTED / NOT_SUPPORTED / INCONCLUSIVE} ## Judge Assessment | | Claude | Review LLM | Final | |---|-------|------|-------| | Verdict | {verdict} | {verdict} | {verdict} | | Confidence | {value} | {value} | {value} | | Evidence strength | {strength} | {strength} | {strength} | ## Key Reasoning {2-3 sentences from Review LLM + Claude synthesis} ## Wiki Changes | Entity | Field | Before | After | |--------|-------|--------|-------| | claims/{slug} | confidence | {old} | {new} | | claims/{slug} | status | {old} | {new} | | ideas/{slug} | status | {old} | {new} | ## Graph Edges Added - experiments/{slug} → claims/{target} (supports/invalidates) ## Concerns {methodological concerns from Review LLM} ## Next Steps - {path-specific suggestions} ## Wiki Growth | Metric | Before | After | Delta | |--------|--------|-------|-------| | Claims updated | — | — | {N} | | Edges | {before} | {after} | +{delta} | | Maturity | {level} | {level} | {unchanged/upgraded} | (Data from comparing `python3 tools/research_wiki.py maturity wiki/ --json` calls at the start of Step 1 and end of Step 4.)
Constraints
- Only process completed experiments: experiments with status != completed are refused; prompt user to use /exp-run first
- Reviewer independence: strictly follow cross-model-review.md — do not send Claude's pre-judgment to Review LLM
- Confidence range 0.0–1.0: updated confidence must not exceed this range
- failure_reason must be specific: the not_supported path's failure_reason cannot be vague (e.g. "experiment failed") — must state the concrete reason
- Do not delete claims: even when not_supported, only challenge or lower confidence; do not delete the claim page. In extreme cases (multiple consistent refutations, confidence → 0), set status to deprecated rather than deleting
- Graph edges via tools/research_wiki.py: do not manually edit edges.jsonl
- Conservative principle: when Claude and Review LLM verdicts disagree, use the more conservative verdict
- Idea status advances only forward: proposed → in_progress → tested → validated/failed, irreversible
- Assess claim using all experiments: consider not just the current experiment but also other experiments on the same claim
Error Handling
- Experiment not found: prompt user to check slug, list candidates in wiki/experiments/ with status=completed
- Experiment not completed: report status, suggest running
/exp-run {slug}or/exp-run {slug} --check - Target claim does not exist: create new claim page (status: proposed, confidence: 0.3), note "auto-created by exp-eval"
- Linked idea does not exist: skip idea update, only update claim, note in report
- Review LLM unavailable: fall back to Claude single-model verdict, note "single-model verdict, cross-model verification unavailable" in report, suggest user confirm later
- Claim was modified by another experiment: read the latest state, make adjustments based on current confidence (do not overwrite other experiments' contributions)
- Results data missing: if the experiment page's Results section is empty, prompt user to run
/exp-run {slug} --checkfirst
Dependencies
Tools(via Bash)
python3 tools/research_wiki.py add-edge wiki/ ...— add graph edgepython3 tools/research_wiki.py rebuild-context-brief wiki/— rebuild query_packpython3 tools/research_wiki.py rebuild-open-questions wiki/— rebuild gap_mappython3 tools/research_wiki.py log wiki/ "<message>"— append log
MCP Servers
mcp__llm-review__chat— Step 2 Review LLM independent verdict
Claude Code Native
Read— read wiki pagesGlob— find other experiments on the same claimEdit— update wiki pages
Shared References
.claude/skills/shared-references/cross-model-review.md— Review LLM independence principle (required reading)
Called by
/researchStage 4 (verdict and iteration stage)- User directly
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |