Skip to content

hoavdc/CodexKit / skills/codexkit-a-b-test-planner

codexkit-a-b-test-planner

Verified

Design rigorous A/B test plans with hypothesis, sample size calculation, Minimum Detectable Effect (MDE), randomization strategy, and decision rules. Includes guardrail metrics and rollout playbook. Use when planning product experiments, conversion optimization, or data-driven feature decisions.

FileKindSize
SKILL.mdskill md5.5 KB
agents/openai.yamlconfig165 B
examples/common-mistakes.mdmarkdown794 B
examples/good-output.mdmarkdown870 B
verification/checklist.mdmarkdown1.4 KB

examples/common-mistakes.md

View on GitHub
# Common Mistakes — A/B Test Planner

> Anti-patterns to watch for when using the A/B Test Planner skill.

## 1. Peeking and stopping early

**What goes wrong:** Checking results daily and stopping when p<0.05 — massively inflates false positives

**How to fix it:** Pre-commit to test duration. If you must peek, use sequential testing methods.

## 2. Testing too many changes at once

**What goes wrong:** New layout + new copy + new colors — can't attribute the effect

**How to fix it:** One change per test. If testing multiple factors, use a proper factorial design.

## 3. Ignoring guardrails

**What goes wrong:** Conversion up 5% but support tickets up 40% — net negative experience

**How to fix it:** Define guardrail metrics upfront. A 'win' must not violate any guardrail.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall