K-Dense-AI/scientific-agent-skills / skills/scientific-critical-thinking
scientific-critical-thinking
VerifiedEvaluates scientific claims and evidence quality. Applies to experimental design validity, biases and confounders, statistical interpretation, evidence grading frameworks (GRADE, Cochrane Risk of Bias), and teaching critical analysis. Supports evidence appraisal and identifying flaws; formal peer review writing belongs to peer-review.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.8 KB |
| references/common_biases.md | markdown | 12.1 KB |
| references/core_capabilities.md | markdown | 16.6 KB |
| references/evidence_hierarchy.md | markdown | 17.0 KB |
| references/experimental_design.md | markdown | 17.8 KB |
| references/logical_fallacies.md | markdown | 18.2 KB |
| references/review_sources.md | markdown | 7.4 KB |
| references/scientific_method.md | markdown | 7.5 KB |
| references/statistical_pitfalls.md | markdown | 20.7 KB |
references/statistical_pitfalls.md
View on GitHub# Common Statistical Pitfalls
Use the estimand, sampling design, and assumptions to judge an analysis; the examples
below are illustrative. Primary guidance is linked in [review_sources.md](review_sources.md).
## P-Value Misinterpretations
### Pitfall 1: P-Value = Probability Hypothesis is True
**Misconception:** p = .05 means 5% chance the null hypothesis is true.
**Reality:** A p-value is a tail probability for the specified test statistic under the
null and the rest of the statistical model, including the sampling/analysis procedure.
It is not a posterior probability that a hypothesis is true.
**Correct interpretation:** "Under this null model and analysis procedure, a test statistic
at least this extreme has probability .05." Selection or unaccounted optional stopping
can invalidate this calibration.
### Pitfall 2: Non-Significant = No Effect
**Misconception:** p > .05 proves there's no effect.
**Reality:** Absence of evidence ≠ evidence of absence. Non-significant results may indicate:
- Insufficient statistical power
- True effect too small to detect
- High variability
- Small sample size
**Better approach:**
- Report confidence intervals
- Compare the interval with effects of practical importance; do not calculate power
from the observed effect to explain this result
- Consider equivalence testing with scientifically justified, prespecified bounds
### Pitfall 3: Significant = Important
**Misconception:** Statistical significance means practical importance.
**Reality:** With large samples, effects too small to matter for the decision can become
"significant." Judge magnitude against a justified practical threshold.
**Better approach:**
- Report effect sizes
- Consider practical significance
- Use confidence intervals
### Pitfall 4: P = .049 vs. P = .051
**Misconception:** These are meaningfully different because one crosses the .05 threshold.
**Reality:** These represent nearly identical evidence. The .05 threshold is arbitrary.
**Better approach:**
- Interpret p-values continuously as model-compatibility summaries, not stand-alone evidence scores
- Report exact p-values
- Consider context and prior evidence
### Pitfall 5: One-Tailed Tests Without Justification
**Misconception:** One-tailed tests are free extra power.
**Reality:** A one-sided test addresses a directional hypothesis; the opposite direction
need not be impossible. Choosing the tail after seeing data invalidates the nominal error rate.
**When appropriate:** Prespecify a directional decision and error criterion (for example,
a justified non-inferiority margin). Still report estimates, uncertainty, and possible harm.
## Multiple Comparisons Problems
### Pitfall 6: Multiple Testing Without Correction
**Problem:** For 20 independent tests of true nulls at alpha .05, the probability of at
least one false rejection is `1 - 0.95**20 = 0.6415` (about 64%). Dependence changes this value.
**Examples:**
- Testing many outcomes
- Testing many subgroups
- Conducting multiple interim analyses
- Testing at multiple time points
**Solutions:**
- Bonferroni correction (divide α by number of tests)
- False Discovery Rate (FDR) control
- Prespecify primary outcome
- Treat exploratory analyses as hypothesis-generating
Define the hypothesis family and error target first. FWER limits the probability of any
false rejection; FDR limits the expected false-discovery proportion (zero when there are
no discoveries). Neither is the probability that a particular discovery is false. A named
primary outcome does not resolve multiplicity across doses, time points, interim looks,
subgroups, or data-selected analysis choices.
### Pitfall 7: Subgroup Analysis Fishing
**Problem:** Testing many subgroups until finding significance.
**Why problematic:**
- Inflates false positive rate
- Often reported without disclosure
- A significant result in women and a non-significant result in men does not establish
a difference between groups; estimate and test the interaction directly
**Solutions:**
- Prespecify subgroups
- Use interaction tests, not separate tests
- Require replication
- Correct for multiple comparisons
### Pitfall 8: Outcome Switching
**Problem:** Analyzing many outcomes, reporting only significant ones.
**Detection signs:**
- Secondary outcomes emphasized
- Incomplete outcome reporting
- Discrepancy between registration and publication
**Solutions:**
- Preregister all outcomes
- Report all planned outcomes
- Distinguish primary from secondary
## Sample Size and Power Issues
### Pitfall 9: Underpowered Studies
**Problem:** Small samples have low probability of detecting true effects.
**Consequences:**
- High false negative rate
- Among selected significant findings, the false-discovery fraction depends on power,
prior prevalence of real effects, bias, and multiplicity; low power alone does not
increase a correctly calibrated test's Type I error above its alpha
- Overestimated effect sizes (when significant)
**Solutions:**
- Conduct a priori power analysis
- Justify power or precision targets for the decision; 80-90% are conventions, not guarantees
- Use a meaningful effect and plausible variance; assess uncertainty in small pilot estimates
### Pitfall 10: Post-Hoc Power Analysis
**Problem:** Calculating "observed power" using the observed effect and the same test is
circular and uninformative. This does not prohibit design sensitivity calculations at an
independently justified effect size.
**Why useless:**
- Non-significant results always have low "post-hoc power"
- It recapitulates the p-value without new information
**Better approach:**
- Calculate confidence intervals
- Plan replication with adequate sample
- Conduct prospective power analysis for future studies
### Pitfall 11: Small Sample Fallacy
**Problem:** Trusting results from very small samples.
**Issues:**
- High sampling variability
- Outliers have large influence
- Distributional approximations may be unreliable; small n alone is not an assumption violation
- Confidence intervals very wide
**Guidelines:**
- There is no universal n = 30 validity threshold; count independent units and inspect precision
- Check assumptions carefully
- Choose methods for the estimand and design; non-parametric methods also have assumptions
- Replicate findings
## Effect Size Misunderstandings
### Pitfall 12: Ignoring Effect Size
**Problem:** Focusing only on significance, not magnitude.
**Why problematic:**
- Significance ≠ importance
- Can't compare across studies
- Doesn't inform practical decisions
**Solutions:**
- Always report effect sizes
- Report effects in interpretable original units or absolute risks; add standardized
measures (Cohen's d, r, η²) when they answer the comparison of interest
- Interpret using field conventions
- Consider minimum clinically important difference
### Pitfall 13: Misinterpreting Standardized Effect Sizes
**Problem:** Treating Cohen's d = 0.5 as "medium" without context.
**Reality:**
- Field-specific norms vary
- Some fields have larger typical effects
- Real-world importance depends on context
**Better approach:**
- Compare to effects in same domain
- Consider practical implications
- Look at raw effect sizes too
### Pitfall 14: Confusing Explained Variance with Importance
**Problem:** "Only explains 5% of variance" = unimportant.
**Reality:**
- A small explained-variance fraction can still matter for an important outcome
- Complex phenomena have many small contributors
- Predictive accuracy ≠ causal importance
**Consideration:** Context matters more than percentage alone.
## Correlation and Causation
### Pitfall 15: Correlation Implies Causation
**Problem:** Inferring causation from correlation.
**Alternative explanations:**
- Reverse causation (B causes A, not A causes B)
- Confounding (C causes both A and B)
- Coincidence
- Selection bias
**Causal appraisal:** Specify the intervention/exposure contrast, population, and outcome.
Assess temporality, exchangeability/confounding, positivity, consistency, selection, and
measurement assumptions. Observational causal inference is possible under defensible
assumptions; randomization does not repair missing outcomes or post-randomization selection.
### Pitfall 16: Ecological Fallacy
**Problem:** Inferring individual-level relationships from group-level data.
**Example:** Countries with more chocolate consumption have more Nobel laureates doesn't mean eating chocolate makes you win Nobels.
**Why problematic:** Group-level correlations may not hold at individual level.
### Pitfall 17: Simpson's Paradox
**Problem:** Trend appears in groups but reverses when combined (or vice versa).
**Example:** Treatment appears worse overall but better in every subgroup.
**Cause:** Different group weights can reverse marginal and conditional associations.
The stratifying variable is not automatically a confounder; it may be a mediator or collider.
**Solution:** Choose adjustment from the causal question and temporal structure, not
from whichever stratification gives the preferred answer.
## Regression and Modeling Pitfalls
### Pitfall 18: Overfitting
**Problem:** Model fits sample data well but doesn't generalize.
**Causes:**
- Too many predictors relative to sample size
- Fitting noise rather than signal
- No cross-validation
**Solutions:**
- Use cross-validation with all learned preprocessing and tuning inside training folds
- Penalized regression (LASSO, ridge)
- Independent test set
- Simpler models
### Pitfall 19: Extrapolation Beyond Data Range
**Problem:** Predicting outside the range of observed data.
**Why dangerous:**
- Relationships may not hold outside observed range
- Increased uncertainty not reflected in predictions
**Solution:** Label extrapolation, justify transport/model assumptions, and assess
sensitivity. In-range predictions can also fail under population or measurement shifts.
### Pitfall 20: Ignoring Model Assumptions
**Problem:** Using statistical tests without checking assumptions.
**Common violations:**
- Incorrect distributional assumptions for the relevant errors/conditional outcomes;
not all parametric models require normally distributed raw data
- Heteroscedasticity (unequal variances)
- Non-independence
- Linearity
- Rank deficiency or unstable coefficient estimation; some predictor correlation is allowed
**Solutions:**
- Check assumptions with diagnostics
- Use robust methods
- Transform data
- Use appropriate non-parametric alternatives
### Pitfall 21: Treating Non-Significant Covariates as Eliminating Confounding
**Problem:** "We controlled for X and it wasn't significant, so it's not a confounder."
**Reality:** Non-significant covariates can still be important confounders. Significance ≠ confounding.
**Solution:** Prespecify an appropriate adjustment set from causal knowledge; do not
select confounders by p-values or automatically adjust for mediators/colliders.
### Pitfall 22: Collinearity Masking Effects
**Problem:** When predictors are highly correlated, true effects may appear non-significant.
**Manifestations:**
- Large standard errors
- Unstable coefficients
- Sign changes when adding/removing variables
**Detection:**
- Variance Inflation Factors (VIF)
- Correlation matrices
**Solutions:**
- Remove redundant predictors
- Combine correlated variables
- Use regularization methods
Regularization may stabilize prediction but does not make a non-identifiable effect
identified by the data. Check design rank, parameter symmetries, profiles/sensitivity,
and dependence on constraints or priors. Good fit and convergence are insufficient.
## Specific Test Misuses
### Pitfall 23: Uncontrolled Pairwise Comparisons
**Problem:** Treating many unadjusted pairwise tests as one confirmatory analysis.
**Correct approach:** Prespecify contrasts and an appropriate familywise or other error
strategy. An omnibus ANOVA is useful for its own question but is not a universal prerequisite
for planned contrasts, and a significant omnibus test does not authorize all unadjusted pairs.
### Pitfall 24: Pearson Correlation for Non-Linear Relationships
**Problem:** Using Pearson's r for curved relationships.
**Why misleading:** r measures linear relationships only.
**Solutions:**
- Check scatterplots first
- Use Spearman's ρ for monotonic relationships
- Consider polynomial or non-linear models
### Pitfall 25: Chi-Square with Small Expected Frequencies
**Problem:** Relying on an asymptotic chi-square approximation with sparse expected counts.
The often-used count-of-five rule is a heuristic, not a universal validity boundary.
**Solutions:**
- Fisher's exact test
- Combine categories only for a prespecified substantive reason, not to obtain significance
- Increase sample size
### Pitfall 26: Paired vs. Independent Tests
**Problem:** Using independent samples test for paired data (or vice versa).
**Why wrong:**
- Ignoring real dependence can misestimate uncertainty in either direction
- Inventing pairs changes the estimand and may discard data or distort uncertainty
**Solution:** Match test to design.
## Confidence Interval Misinterpretations
### Pitfall 27: 95% CI = 95% Probability True Value Inside
**Misconception:** "95% chance the true value is in this interval."
**Reality:** For a correctly calibrated frequentist 95% interval procedure under its
assumptions, 95% of intervals across repetitions cover the fixed parameter. An individual
interval does not assign a 95% posterior probability to that parameter.
**Better reporting:** Give the estimate, interval, confidence level, assumptions, and
whether the interval excludes effects important to the scientific decision.
### Pitfall 28: Overlapping CIs = No Difference
**Problem:** Assuming overlapping confidence intervals mean no significant difference.
**Reality:** Overlapping CIs are less stringent than difference tests. Two CIs can overlap while the difference between groups is significant.
**Guideline:** Estimate the contrast and its interval directly, accounting for covariance.
Neither overlap heuristic substitutes for that calculation.
### Pitfall 29: Ignoring CI Width
**Problem:** Focusing only on whether CI includes zero, not precision.
**Why important:** Wide CIs indicate high uncertainty. "Significant" effects with huge CIs are less convincing.
**Consider:** Both significance and precision.
## Bayesian vs. Frequentist Confusions
### Pitfall 30: Mixing Bayesian and Frequentist Interpretations
**Problem:** Making Bayesian statements from frequentist analyses.
**Examples:**
- "Probability hypothesis is true" (Bayesian) from p-value (frequentist)
- "Evidence for no meaningful effect" from a non-significant test alone; frequentist
equivalence tests can address prespecified bounds, not prove an exact point null
**Solution:**
- Be clear about framework
- Use Bayesian methods for Bayesian questions
- For Bayes factors, specify competing models and priors and examine prior sensitivity
### Pitfall 31: Ignoring Prior Probability
**Problem:** Treating all hypotheses as equally likely initially.
**Reality:** Extraordinary claims need extraordinary evidence. Prior plausibility matters.
**Consider:**
- Plausibility given existing knowledge
- Mechanism plausibility
- Base rates
## Data Transformation Issues
### Pitfall 32: Dichotomizing Continuous Variables
**Problem:** Splitting continuous variables at arbitrary cutoffs.
**Consequences:**
- Loss of information and power
- Arbitrary distinctions
- Discarding individual differences
**Exceptions:** Clinically meaningful cutoffs with strong justification.
**Better:** Model the continuous relationship, including nonlinear terms when justified;
adding arbitrary categories still loses information.
### Pitfall 33: Trying Multiple Transformations
**Problem:** Testing many transformations until finding significance.
**Why problematic:** Inflates Type I error, is a form of p-hacking.
**Better approach:**
- Prespecify transformations
- Use theory-driven transformations
- Correct for multiple testing if exploring
## Missing Data Problems
### Pitfall 34: Listwise Deletion by Default
**Problem:** Automatically deleting all cases with any missing data.
**Consequences:**
- Reduced power
- Potential bias if data not missing completely at random (MCAR)
**Better approaches:**
- Multiple imputation
- Maximum likelihood methods
- Analyze missingness patterns
### Pitfall 35: Ignoring Missing Data Mechanisms
**Problem:** Not considering why data are missing.
**Types:**
- MCAR: missingness independent of observed and unobserved data; deletion can lose precision
- MAR: conditional on the observed variables used, missingness is independent of missing
values; imputation/likelihood methods still require correctly specified, compatible models
- MNAR: residual dependence on missing values; examine plausible sensitivity scenarios
**Solution:** Record reasons, timing, and assumptions. Observed missingness patterns cannot
establish MAR versus MNAR. Pool imputation uncertainty; single mean imputation is not a fix.
## Publication and Reporting Issues
### Pitfall 36: Selective Reporting
**Problem:** Only reporting significant results or favorable analyses.
**Consequences:**
- Literature appears more consistent than reality
- Meta-analyses biased
- Wasted research effort
**Solutions:**
- Preregistration
- Report all analyses
- Use reporting guidelines (CONSORT, PRISMA, etc.)
### Pitfall 37: Rounding to p < .05
**Problem:** Reporting exact p-values selectively (e.g., p = .049 but p < .05 for .051).
**Why problematic:** Obscures values near threshold, enables p-hacking detection evasion.
**Better:** Always report exact p-values.
### Pitfall 38: No Data Sharing
**Problem:** Not making data available for verification or reanalysis.
**Consequences:**
- Can't verify results
- Some independent checks or individual-participant-data analyses may be impossible;
published aggregate estimates can still support an appropriate meta-analysis
- Hinders scientific progress
**Best practice:** Share data unless privacy concerns prohibit.
## Cross-Validation and Generalization
### Pitfall 39: No Cross-Validation
**Problem:** Testing model on same data used to build it.
**Consequence:** Overly optimistic performance estimates.
**Solutions:**
- Split data (train/test)
- K-fold cross-validation
- Independent validation sample
### Pitfall 40: Data Leakage
**Problem:** Information from test set leaking into training.
**Examples:**
- Normalizing before splitting
- Feature selection on full dataset
- Including temporal information
**Consequence:** Inflated performance metrics.
**Prevention:** Fit learned preprocessing only on the training portion of each fold; tune
within that boundary and reserve an untouched evaluation set. Split by independent subject,
site, batch, or time as the deployment question requires; technical replicates must not leak.
## Meta-Analysis Pitfalls
### Pitfall 41: Apples and Oranges
**Problem:** Combining studies with different designs, populations, or measures.
**Balance:** Need homogeneity but also comprehensiveness.
**Solutions:**
- Clear inclusion criteria
- Subgroup analyses
- Meta-regression for moderators
### Pitfall 42: Ignoring Publication Bias
**Problem:** Published studies overrepresent significant results.
**Consequences:** Overestimated effects in meta-analyses.
**Assessment:** Compare protocols/registries and reported outcomes; seek eligible unpublished
results. Funnel asymmetry indicates small-study effects, not necessarily publication bias.
Asymmetry tests generally need at least ten studies with varying precision and a method
appropriate to the effect measure; a negative test cannot exclude missing evidence.
**Sensitivity analysis:** Trim-and-fill, selection models, PET-PEESE, and p-curve have
different assumptions and targets. Do not describe adjusted estimates as recovered truth
or treat fail-safe N as a reliability certificate.
### Pitfall 43: Pseudoreplication and Changing Denominators
Repeated measurements, cells from one donor, or wells from one culture are not automatically
independent biological replicates. Identify assignment and sampling units; use an appropriate
hierarchical/repeated-measures analysis or justified aggregation. Report independent-unit n
separately from measurement count. A mean of group percentages differs from the pooled
numerator/denominator when group sizes differ; state weighting, evaluable counts, and exclusions.
## General Best Practices
1. **Preregister studies** - Distinguish confirmatory from exploratory
2. **Report transparently** - All analyses, not just significant ones
3. **Check assumptions** - Don't blindly apply tests
4. **Use appropriate tests** - Match test to data and design
5. **Report effect sizes** - Not just p-values
6. **Consider practical significance** - Not just statistical
7. **Replicate findings** - One study is rarely definitive
8. **Share data and code** - Enable verification
9. **Use confidence intervals** - Show uncertainty
10. **Think causally carefully** - Most research is correlational
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |