borghei/Claude-Skills / product-team/ab-test-setup
ab-test-setup
VerifiedDesign and run statistically rigorous A/B tests and experiments. Use when planning experiments, calculating sample sizes, designing test variants, selecting metrics, analyzing results, or when someone says "let's test that."
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 20.7 KB |
| evals/README.md | markdown | 1.0 KB |
| evals/grader.py | script | 5.9 KB |
| evals/runner.py | script | 7.7 KB |
| evals/test_cases.json | config | 3.5 KB |
| scripts/experiment_analyzer.pyexecutable | script | 9.6 KB |
| scripts/experiment_planner.pyexecutable | script | 14.0 KB |
| scripts/sample_size_calculator.pyexecutable | script | 9.6 KB |
evals/README.md
View on GitHub# Evals — ab-test-setup
Drop-in eval harness for the `ab-test-setup` skill. See `../../../templates/evals-template/README.md` for the general pattern.
## What these cases cover
| Case | Tests |
|------|-------|
| `sample-size-basics` | Skill computes (or asks for) baseline, MDE, power, alpha; produces a realistic sample size |
| `no-peeking` | Skill refuses early stopping on interim significance and names the statistical reason |
| `metric-selection-multiple-primary` | Skill enforces a single primary metric / OEC and demotes the rest to guardrails |
| `underpowered-test-warning` | Skill flags an n=500/arm test as underpowered rather than greenlighting it |
## Running
```bash
# Static validation (no model needed)
python evals/runner.py --skill ../
# Graded mode (after an external harness captures outputs)
python evals/grader.py --candidate candidate.json --format text
```
`runner.py` and `grader.py` are copied from `templates/evals-template/evals/` and not modified for this skill — keep them in sync with the template.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |