alirezarezvani/claude-skills / product-team/skills/experiment-designer
experiment-designer
VerifiedPlan A/B tests and product experiments: hypotheses, sample size, priority, reading results. Use whenever the user asks if a test won, how long to run it or what to test.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.1 KB |
| references/experiment-playbook.md | markdown | 1.9 KB |
| references/statistics-reference.md | markdown | 1.5 KB |
| scripts/sample_size_calculator.pyexecutable | script | 3.0 KB |
references/statistics-reference.md
View on GitHub# Statistics Reference for Product Managers
## p-value
The p-value is the probability of observing data at least as extreme as yours if there were no true effect.
- Small p-value means data is less consistent with "no effect".
- It does not tell you the probability that the variant is best.
## Confidence Interval (CI)
A CI gives a plausible range for the true effect size.
- Narrow interval: more precise estimate.
- Wide interval: uncertain estimate.
- If CI includes zero (or no-effect), directional confidence is weak.
## Minimum Detectable Effect (MDE)
The smallest effect worth detecting.
- Set MDE by business value threshold, not wishful optimism.
- Smaller MDE requires larger sample size.
## Statistical Power
Power is the probability of detecting a true effect of at least MDE.
- Common target: 80% (0.8)
- Higher power increases sample requirements.
## Type I and Type II Errors
- Type I (false positive): claim effect when none exists (controlled by alpha).
- Type II (false negative): miss a real effect (controlled by power).
## Practical Significance
An effect can be statistically significant but too small to matter.
Always ask:
- Does the effect clear implementation cost?
- Does it move strategic KPIs materially?
## Power Analysis Inputs
For conversion experiments (two proportions):
- Baseline conversion rate
- MDE (absolute points or relative uplift)
- Alpha (e.g., 0.05)
- Power (e.g., 0.8)
Output:
- Required sample size per variant
- Total sample size
- Approximate runtime based on traffic volume
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |