dvy1987/agent-loom / .agents/skills/experiment-spec
experiment-spec
VerifiedWrite a rigorous, decision-grade experiment spec — falsifiable hypothesis, primary metric, guardrails, randomisation unit, exposure definition, method (A/B, holdout, switchback, quasi-experiment, MAB), MDE/duration plan, peek policy, validity threats, and pre-committed decision rule. Platform-agnostic. Load when the user has a candidate experiment and needs to spec it before launch, or says "spec this experiment", "write the test plan", "design this A/B test", "what's the hypothesis", "how big a sample do we need", "how long should we run this", "define the metrics for this test", or when the experimentation orchestrator routes here.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.1 KB |
| references/examples.md | markdown | 2.2 KB |
| references/mde-heuristics.md | markdown | 2.9 KB |
| references/spec-template.md | markdown | 3.2 KB |
| references/validity-threats.md | markdown | 5.0 KB |
references/validity-threats.md
View on GitHub# Validity Threats
Catalogue of the failure modes that silently invalidate experiment results. List the ones relevant to the surface in the spec's `Validity Threats` section, with the planned mitigation for each.
---
## 1. Sample Ratio Mismatch (SRM)
**What:** observed traffic split deviates from the intended split (e.g. expected 50/50, observed 52/48). Signals broken randomisation, biased traffic routing, or logging failures.
**Impact:** invalidates results. SRM-positive tests routinely produce phantom wins.
**Mitigation:** chi-squared SRM check at every ramp gate and at readout. Block analysis if p-value < 0.001 sustained.
**Apply to:** every test, no exceptions.
---
## 2. Novelty Effect
**What:** users react to "new" — not "better". The lift is real but temporary.
**Impact:** initial weeks show inflated lift; effect decays or reverses long-term.
**Mitigation:**
- Phase-2 holdout window (4+ weeks post-ship) to validate persistence.
- Exclude first 24–48h of exposure from the analysis window if traffic permits.
- Larger samples to allow within-test detection of decay.
**Apply to:** redesigns, onboarding changes, prominent UI changes, gamification.
---
## 3. Primacy Effect
**What:** the inverse of novelty — users get worse before they get better because they were trained on the old experience.
**Impact:** new variant looks worse short-term but is actually better long-term.
**Mitigation:** longer test duration; segment by user tenure; planned re-evaluation window.
**Apply to:** flow restructures, shortcut removals, navigation changes.
---
## 4. Interference (SUTVA Violation)
**What:** one unit's treatment affects another unit's outcome. The Stable Unit Treatment Value Assumption fails.
**Impact:** A/B is invalid because control "leaks" treatment.
**Mitigation:**
- **Switchback** (alternate treatment on/off across time windows for the whole population).
- **Cluster-randomisation** by region/group/account.
**Apply to:** marketplaces (buyer/seller), feeds with shared inventory, scheduling/dispatch, social ranking, recommendations.
---
## 5. Contamination
**What:** units assigned to one variant are exposed to the other (shared device, shared account, multi-tab sessions, mis-routed traffic).
**Impact:** dilutes the measured effect; can flip the sign in extreme cases.
**Mitigation:** server-side assignment with deterministic hashing; consistent unit definition across surfaces; explicit handling of multi-device users.
**Apply to:** consumer SaaS with multi-device users, B2B shared accounts, public unauth flows.
---
## 6. Channel-Mix Confound
**What:** upstream traffic source allocation shifts during the test. Variant differences reflect channel mix, not treatment effect.
**Impact:** ghost wins and ghost losses.
**Mitigation:**
- Restrict population to a stable channel (e.g. organic only).
- Stratify analysis by channel.
- Watch for paid campaign launches/changes during the test window.
**Apply to:** landing pages, signup flows, anything reachable from paid channels.
---
## 7. Instrumentation Drift
**What:** event firing rate or property structure changes mid-test (deploy of unrelated code, SDK upgrade, schema change).
**Impact:** metric calculations break across the test boundary; analysis becomes unreliable.
**Mitigation:** lock instrumentation during the test; alert on event-rate anomalies; re-validate exposure event at every ramp gate.
**Apply to:** every test, especially those running across active deploy cycles.
---
## 8. Multiple Comparisons
**What:** running many secondary metrics (or many variants) inflates the false-positive rate without correction.
**Impact:** spurious "significant" findings that don't reproduce.
**Mitigation:**
- Pre-register primary + ≤3 secondaries.
- Apply Bonferroni or Benjamini-Hochberg correction for additional metrics.
- Treat all post-hoc findings as exploratory.
**Apply to:** tests with 3+ secondaries or 3+ variants.
---
## 9. Selection Bias
**What:** the population that triggers the experiment is non-random (e.g. only users who reached step 3 of onboarding see the test; their characteristics differ from the full population).
**Impact:** treatment effect doesn't generalise.
**Mitigation:** trigger-based analysis (only compare units who actually were exposed); document the exposed sub-population explicitly.
**Apply to:** mid-funnel surfaces, conditional features.
---
## 10. Long-Term Effect Lag
**What:** the metric that matters (revenue, retention, LTV) lags the test window.
**Impact:** short-loop proxy moves but headline metric doesn't follow.
**Mitigation:** declare a short-loop proxy AND a long-loop final metric; commit to a Phase-2 holdout for the long-loop validation.
**Apply to:** monetisation tests, retention tests, anything with revenue lag > test duration.
---
## How to Use This File
1. Read each section.
2. Mark the threats that apply to your surface.
3. For each marked threat, write the mitigation into the spec.
4. If a threat applies but cannot be mitigated, downgrade the test to Directional Exploration or change the method.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |