Skip to content

hoavdc/CodexKit / skills/codexkit-a-b-test-planner

codexkit-a-b-test-planner

Verified

Design rigorous A/B test plans with hypothesis, sample size calculation, Minimum Detectable Effect (MDE), randomization strategy, and decision rules. Includes guardrail metrics and rollout playbook. Use when planning product experiments, conversion optimization, or data-driven feature decisions.

FileKindSize
SKILL.mdskill md5.5 KB
agents/openai.yamlconfig165 B
examples/common-mistakes.mdmarkdown794 B
examples/good-output.mdmarkdown870 B
verification/checklist.mdmarkdown1.4 KB

verification/checklist.md

View on GitHub
# 4C Verification Checklist — A/B Test Planner

> Use this checklist after generating output. Answer each question honestly.

## Correctness — Is it right?

- [ ] Sample size calculation uses the correct MDE, baseline rate, and significance level
- [ ] Randomization unit matches the analysis unit (user-level, not session-level for most tests)
- [ ] Primary metric is defined before the test starts — no post-hoc metric shopping

## Completeness — Is it done?

- [ ] Hypothesis, primary metric, guardrail metrics, and test duration are all defined
- [ ] Power analysis determines minimum sample size and test duration
- [ ] Rollback criteria and escalation path are documented

## Context-fit — Does it match?

- [ ] Test duration accounts for weekly/monthly traffic patterns
- [ ] MDE is meaningful from a business perspective, not just statistically detectable
- [ ] Guardrail metrics prevent optimizing the primary metric at the expense of user experience

## Consequence — Is it safe to use?

- [ ] If the test runs 20% longer than planned, what's the cost?
- [ ] If we ship the winner without statistical significance, what's the false-positive risk?
- [ ] If the variation loses, is there a learning that informs the next test?

---

**Final question:** _If this output were used immediately in production, what is the single biggest risk?_

Write the answer here before considering the skill output complete.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 321kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 321kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 321kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall