MaarkssDN/Birthub-360 / .agents/skills/ab-test-validity-checklist
ab-test-validity-checklist
VerifiedUse when launching, auditing or governing an A/B test or online experiment design. Builds a per-experiment checklist and sign-off: pre-registered metric, power and guardrails, randomization unit and sample ratio mismatch checks, assignment, telemetry, peeking rules and novelty effects.
| File | Kind | Size |
|---|---|---|
| .tessl-plugin/plugin.json | config | 429 B |
| SKILL.md | skill md | 7.7 KB |
| references/checklist-template.md | markdown | 1.7 KB |
references/checklist-template.md
View on GitHub# Validity checklist and sign-off template
Copy to `experiments/<id>/checklist.md` at pre-registration and fill in
each section as the experiment moves through
[SKILL.md](../SKILL.md) Steps 1-7. Every ticked box needs its evidence
next to it (a number, a query link, or a calculation). The SRM
threshold follows Microsoft ExP's p < 0.0005 ([ExP SRM article][srm]).
```markdown
# Experiment <id> - Validity Checklist
## Pre-registration (signed by: <owner>, date: <YYYY-MM-DD>)
- [ ] OEC declared: <metric>
- [ ] Power calc: N=<X>, alpha=0.05, beta=0.20, MDE=<Y>%
- [ ] Guardrails declared: <list with thresholds>
- [ ] Randomization unit: <user_id / device_id>
- [ ] Allocation: <50/50>
- [ ] Look schedule: <Pocock 5 looks at days 2,4,7,10,14>
- [ ] Stop-early rules: <on OEC reaching alpha-threshold>
## During experiment
- [ ] SRM check: chi-square p >= 0.0005 ([result: <p>])
- [ ] Assignment integrity tests passing
- [ ] Telemetry validated
## Post-experiment (signed by: <reviewer>, date: <YYYY-MM-DD>)
- [ ] Pre-registration honoured (no scope changes)
- [ ] SRM final check: p >= 0.0005 ([result])
- [ ] OEC significant (p=<X>; method: <Pocock>)
- [ ] All guardrails within thresholds:
- api_p95_latency: +<X>% / +<Y>ms - <status>
- dau: <X>% - <status>
- [ ] Multiple-comparison adjusted (method: <Bonferroni / BH>)
- [ ] Novelty assessment: effect persists week 2+? <yes / no>
- [ ] Segment stability: direction consistent? <yes / no>
- [ ] Trust metric stable? <yes / no>
## Ship decision: <ship / no-ship / extend>
Reasoning: <one paragraph>
Sign-off: <name>, <date>
```
The reviewer who signs the post-experiment section is not the
experiment owner.
[srm]: https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |