Skip to content

dvy1987/agent-loom / .agents/skills/experiment-spec

experiment-spec

Verified

Write a rigorous, decision-grade experiment spec — falsifiable hypothesis, primary metric, guardrails, randomisation unit, exposure definition, method (A/B, holdout, switchback, quasi-experiment, MAB), MDE/duration plan, peek policy, validity threats, and pre-committed decision rule. Platform-agnostic. Load when the user has a candidate experiment and needs to spec it before launch, or says "spec this experiment", "write the test plan", "design this A/B test", "what's the hypothesis", "how big a sample do we need", "how long should we run this", "define the metrics for this test", or when the experimentation orchestrator routes here.

FileKindSize
SKILL.mdskill md10.1 KB
references/examples.mdmarkdown2.2 KB
references/mde-heuristics.mdmarkdown2.9 KB
references/spec-template.mdmarkdown3.2 KB
references/validity-threats.mdmarkdown5.0 KB

references/examples.md

View on GitHub
# Experiment Spec — Full Worked Examples

Skill: `experiment-spec` | Enriched from SKILL.md (improve-skills pass, SKIP_RESEARCH).

## Example 1 — Step-by-step execution

**Input:** "Run `experiment-spec` on [concrete task]"

**Agent actions:**
1. Frame the Hypothesis
2. Declare Decision Class
3. Choose the Method
4. Define Unit and Exposure
5. Define Metrics
6. Sample Size & Duration
7. List Validity Threats
8. Decision Rule + Peek Policy

**Impact Report shape:**
```
Spec written: docs/experiments/specs/YYYY-MM-DD-<slug>-spec.md
Decision class: [Causal | Directional | Instrumentation]
Method: [A/B | Holdout | Switchback | Quasi | MAB]
Primary metric: [name, direction, MDE]
Guardrails: [list]
Sample plan: [N per arm × duration weeks]
Decision rule: [one-liner]
Validity threats listed: [count]
Status: [READY-TO-LAUNCH | DOWNGRADED-TO-DIRECTIONAL | BLOCKED-INSUFFICIENT-POWER]
```

## Example 2 — Anti-skip (rationalization defense)

**Input:** Agent tries to skip a gate

| Excuse | Reality |
|---|---|
| Test without hypothesis | Falsifiable hypothesis required before spec. |
| Peek until significant | Peek policy must be pre-committed in spec. |
| Any metric goes | Primary + guardrail metrics defined up front. |
| Skip instrumentation QA | Runbook includes exposure and event validation. |

## Example 3 — Gotcha application

**Input:** Task hits a non-obvious edge case

**Apply:**
- **MDE is relative, not absolute.** "5% lift" almost always means 5% **relative** to baseline (4.0% → 4.2%) — not 5 percentage points (4.0% → 9.0%). Stating MDE without the unit is the #1 source of post-launch surprise about sample size.
- **The if-clause IS the spec.** A spec without "if it does, we will [decision]" is not falsifiable — it's an aspiration. Refuse to finalise until the if-clause exists.
- **Exposure event ≠ flag fetch.** The spec must define exposure as the moment the user *sees* the variant. Conflating the two means SRM checks are meaningless and the readout will silently fail.
- **Duration must cover whole-week multiples.** Day-of-week effects (e.g., weekend signups) bias short tests. Round up to 7, 14, or 21 days; a "10-day test" almost always misrepresents weekly seasonality.

---

See `SKILL.md` for hard rules and verification checklist.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall