hoavdc/CodexKit / skills/codexkit-a-b-test-planner
codexkit-a-b-test-planner
VerifiedDesign rigorous A/B test plans with hypothesis, sample size calculation, Minimum Detectable Effect (MDE), randomization strategy, and decision rules. Includes guardrail metrics and rollout playbook. Use when planning product experiments, conversion optimization, or data-driven feature decisions.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.5 KB |
| agents/openai.yaml | config | 165 B |
| examples/common-mistakes.md | markdown | 794 B |
| examples/good-output.md | markdown | 870 B |
| verification/checklist.md | markdown | 1.4 KB |
verification/checklist.md
View on GitHub# 4C Verification Checklist — A/B Test Planner
> Use this checklist after generating output. Answer each question honestly.
## Correctness — Is it right?
- [ ] Sample size calculation uses the correct MDE, baseline rate, and significance level
- [ ] Randomization unit matches the analysis unit (user-level, not session-level for most tests)
- [ ] Primary metric is defined before the test starts — no post-hoc metric shopping
## Completeness — Is it done?
- [ ] Hypothesis, primary metric, guardrail metrics, and test duration are all defined
- [ ] Power analysis determines minimum sample size and test duration
- [ ] Rollback criteria and escalation path are documented
## Context-fit — Does it match?
- [ ] Test duration accounts for weekly/monthly traffic patterns
- [ ] MDE is meaningful from a business perspective, not just statistically detectable
- [ ] Guardrail metrics prevent optimizing the primary metric at the expense of user experience
## Consequence — Is it safe to use?
- [ ] If the test runs 20% longer than planned, what's the cost?
- [ ] If we ship the winner without statistical significance, what's the false-positive risk?
- [ ] If the variation loses, is there a learning that informs the next test?
---
**Final question:** _If this output were used immediately in production, what is the single biggest risk?_
Write the answer here before considering the skill output complete.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |