dvy1987/agent-loom / .agents/skills/experiment-spec
experiment-spec
VerifiedWrite a rigorous, decision-grade experiment spec — falsifiable hypothesis, primary metric, guardrails, randomisation unit, exposure definition, method (A/B, holdout, switchback, quasi-experiment, MAB), MDE/duration plan, peek policy, validity threats, and pre-committed decision rule. Platform-agnostic. Load when the user has a candidate experiment and needs to spec it before launch, or says "spec this experiment", "write the test plan", "design this A/B test", "what's the hypothesis", "how big a sample do we need", "how long should we run this", "define the metrics for this test", or when the experimentation orchestrator routes here.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.1 KB |
| references/examples.md | markdown | 2.2 KB |
| references/mde-heuristics.md | markdown | 2.9 KB |
| references/spec-template.md | markdown | 3.2 KB |
| references/validity-threats.md | markdown | 5.0 KB |
references/examples.md
View on GitHub# Experiment Spec — Full Worked Examples
Skill: `experiment-spec` | Enriched from SKILL.md (improve-skills pass, SKIP_RESEARCH).
## Example 1 — Step-by-step execution
**Input:** "Run `experiment-spec` on [concrete task]"
**Agent actions:**
1. Frame the Hypothesis
2. Declare Decision Class
3. Choose the Method
4. Define Unit and Exposure
5. Define Metrics
6. Sample Size & Duration
7. List Validity Threats
8. Decision Rule + Peek Policy
**Impact Report shape:**
```
Spec written: docs/experiments/specs/YYYY-MM-DD-<slug>-spec.md
Decision class: [Causal | Directional | Instrumentation]
Method: [A/B | Holdout | Switchback | Quasi | MAB]
Primary metric: [name, direction, MDE]
Guardrails: [list]
Sample plan: [N per arm × duration weeks]
Decision rule: [one-liner]
Validity threats listed: [count]
Status: [READY-TO-LAUNCH | DOWNGRADED-TO-DIRECTIONAL | BLOCKED-INSUFFICIENT-POWER]
```
## Example 2 — Anti-skip (rationalization defense)
**Input:** Agent tries to skip a gate
| Excuse | Reality |
|---|---|
| Test without hypothesis | Falsifiable hypothesis required before spec. |
| Peek until significant | Peek policy must be pre-committed in spec. |
| Any metric goes | Primary + guardrail metrics defined up front. |
| Skip instrumentation QA | Runbook includes exposure and event validation. |
## Example 3 — Gotcha application
**Input:** Task hits a non-obvious edge case
**Apply:**
- **MDE is relative, not absolute.** "5% lift" almost always means 5% **relative** to baseline (4.0% → 4.2%) — not 5 percentage points (4.0% → 9.0%). Stating MDE without the unit is the #1 source of post-launch surprise about sample size.
- **The if-clause IS the spec.** A spec without "if it does, we will [decision]" is not falsifiable — it's an aspiration. Refuse to finalise until the if-clause exists.
- **Exposure event ≠ flag fetch.** The spec must define exposure as the moment the user *sees* the variant. Conflating the two means SRM checks are meaningless and the readout will silently fail.
- **Duration must cover whole-week multiples.** Day-of-week effects (e.g., weekend signups) bias short tests. Round up to 7, 14, or 21 days; a "10-day test" almost always misrepresents weekly seasonality.
---
See `SKILL.md` for hard rules and verification checklist.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |