dvy1987/agent-loom / .agents/skills/experiment-runbook
experiment-runbook
VerifiedTranslate an approved experiment spec into a launch runbook — platform binding (PostHog primary), feature flag setup, assignment unit, exposure event definition, instrumentation QA, dashboard wiring, ramp plan, monitoring, and rollback procedure. Platform-agnostic core with one strong PostHog adapter shipped; GrowthBook, Statsig, LaunchDarkly, Optimizely, and Eppo documented as a single mapping table the user adapts. Load when a spec is approved and ready to launch, or when the user says "set up the experiment", "wire this up in PostHog", "implement the test", "create the runbook", "launch checklist for this test", or when the experimentation orchestrator routes here.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.7 KB |
| references/examples.md | markdown | 2.2 KB |
| references/launch-qa-checklist.md | markdown | 3.9 KB |
| references/posthog-binding.md | markdown | 4.9 KB |
| references/vendor-mapping.md | markdown | 4.4 KB |
references/examples.md
View on GitHub# Experiment Runbook — Full Worked Examples
Skill: `experiment-runbook` | Enriched from SKILL.md (improve-skills pass, SKIP_RESEARCH).
## Example 1 — Step-by-step execution
**Input:** "Run `experiment-runbook` on [concrete task]"
**Agent actions:**
1. Read the Spec
2. Define Platform Binding
3. Define Assignment
4. Define Exposure Event
5. Wire Dashboards & Alerts
6. Ramp Plan
7. Pre-Launch QA
8. Rollback Procedure
**Impact Report shape:**
```
Runbook written: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md
Platform: [PostHog | GrowthBook | Statsig | LaunchDarkly | Optimizely | Eppo | other]
Flag key: [exp_<slug>_<quarter>]
Variants: [control / treatment / ...]
Allocation: [50/50 | 90/10 holdout | ...]
Exposure event: [event_name]
Ramp plan: [1% → 5% → 50% with gates]
QA checklist: [N/N pass]
Rollback: [documented yes/no]
Status: [READY-TO-LAUNCH | BLOCKED-QA-FAIL | BLOCKED-MISSING-SPEC]
```
## Example 2 — Anti-skip (rationalization defense)
**Input:** Agent tries to skip a gate
| Excuse | Reality |
|---|---|
| Test without hypothesis | Falsifiable hypothesis required before spec. |
| Peek until significant | Peek policy must be pre-committed in spec. |
| Any metric goes | Primary + guardrail metrics defined up front. |
## Example 3 — Gotcha application
**Input:** Task hits a non-obvious edge case
**Apply:**
- **Exposure event must fire when the variant renders, not when the flag is fetched.** Server-side renders and async surfaces commonly log exposure before the variant actually loads — this guarantees SRM noise and unanalysable results.
- **`$feature_flag_called` is PostHog's exposure event.** Capturing a custom event without `$feature_flag` and `$feature_flag_response` properties means PostHog's experiment UI cannot match exposures to assignments.
- **Person-property assignment for B2B accounts is wrong.** Use group-property (account-level) — otherwise users on the same account land in different variants and contaminate the test.
- **Salt the hash and lock it.** A drifting salt re-randomises mid-test; the same user gets different variants on different sessions, destroying causal interpretation. PostHog does this for you; verify other vendors.
---
See `SKILL.md` for hard rules and verification checklist.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |