Skip to content

dvy1987/agent-loom / .agents/skills/experiment-readout

experiment-readout

Verified

Analyse experiment results, run validity checks (SRM, exposure parity, data integrity, novelty/primacy), interpret causally, make a ship/iterate/kill decision against the pre-declared rule, and append to cumulative learnings. Forces honest readouts — strips significance claims from underpowered or peek-violating tests; never lets directional results masquerade as causal wins. Load when results exist, or when the user says "read out this experiment", "analyse the test", "did the test win", "interpret the results", "what did we learn", "ship or kill", or when the experimentation orchestrator routes here.

FileKindSize
SKILL.mdskill md10.5 KB
references/examples.mdmarkdown2.6 KB
references/learnings-format.mdmarkdown4.7 KB
references/readout-template.mdmarkdown3.5 KB
references/validity-checks.mdmarkdown4.4 KB

references/examples.md

View on GitHub
# Experiment Readout — Full Worked Examples

Skill: `experiment-readout` | Enriched from SKILL.md (improve-skills pass, SKIP_RESEARCH).

## Example 1 — Step-by-step execution

**Input:** "Run `experiment-readout` on [concrete task]"

**Agent actions:**
1. Pre-Flight Validity Checks (Blocking)
2. Compute Primary Metric Effect
3. Compute Guardrail Metrics
4. Apply the Decision Rule
5. Novelty / Long-Term Check (Conditional)
6. Segment & Exploratory Analysis (Optional, Tagged)
7. Write the Analysis File
8. Append to Learnings

**Impact Report shape:**
```
Analysis: docs/experiments/analyses/YYYY-MM-DD-<slug>-analysis.md
Validity: SRM=[PASS/FAIL] | Exposure parity=[ok/skew] | Event rate=[stable/drift]
Sample sufficiency: [adequate / underpowered]
Primary effect: [point estimate, 95% CI, p-value or posterior]
Guardrails: [N held / N breached]
Decision: [SHIP | ITERATE | KILL | INCONCLUSIVE | SRM-FAIL]
Decision-rule match: [literal condition met]
Novelty / long-term: [check result, holdout required yes/no]
Learnings entry: [appended yes]
Downstream: [prd-writing | architectural-decision-log | reality-check | none]
```

## Example 2 — Anti-skip (rationalization defense)

**Input:** Agent tries to skip a gate

| Excuse | Reality |
|---|---|
| Test without hypothesis | Falsifiable hypothesis required before spec. |
| Peek until significant | Peek policy must be pre-committed in spec. |
| Any metric goes | Primary + guardrail metrics defined up front. |
| Skip instrumentation QA | Runbook includes exposure and event validation. |

## Example 3 — Gotcha application

**Input:** Task hits a non-obvious edge case

**Apply:**
- **SRM is the silent killer of trust.** A failed SRM invalidates everything downstream — not "with caveats", entirely. The chi-squared check runs first and is blocking. Phantom wins almost always trace to broken randomisation, not real lift.
- **"Significant" is forbidden vocabulary for Directional and Instrumentation tests.** Watch for it slipping in via stakeholder summaries; strip it from every readout that wasn't pre-declared Causal with adequate power.
- **Confidence intervals that exclude zero ≠ a "win" if a guardrail breached.** The decision rule is conjunctive — primary AND guardrails. A primary lift with a guardrail breach is KILL, not "ship with watchlist".
- **Exposure parity gaps point at instrumentation, not user behaviour.** A 5%+ gap between assigned and exposed is almost always an asynchronous render or flag-fetch-before-render issue. Investigate before reporting metrics, not after.

---

See `SKILL.md` for hard rules and verification checklist.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall