K-Dense-AI/scientific-agent-skills / skills/scientific-critical-thinking
scientific-critical-thinking
VerifiedEvaluates scientific claims and evidence quality. Applies to experimental design validity, biases and confounders, statistical interpretation, evidence grading frameworks (GRADE, Cochrane Risk of Bias), and teaching critical analysis. Supports evidence appraisal and identifying flaws; formal peer review writing belongs to peer-review.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.8 KB |
| references/common_biases.md | markdown | 12.1 KB |
| references/core_capabilities.md | markdown | 16.6 KB |
| references/evidence_hierarchy.md | markdown | 17.0 KB |
| references/experimental_design.md | markdown | 17.8 KB |
| references/logical_fallacies.md | markdown | 18.2 KB |
| references/review_sources.md | markdown | 7.4 KB |
| references/scientific_method.md | markdown | 7.5 KB |
| references/statistical_pitfalls.md | markdown | 20.7 KB |
references/experimental_design.md
View on GitHub# Experimental Design Checklist
Apply items to the actual design rather than treating checked boxes as a quality score.
Framework versions and primary guidance: [review_sources.md](review_sources.md).
## Research Question Formulation
### Is the Question Well-Formed?
- [ ] **Specific:** Clearly defined variables and relationships
- [ ] **Answerable:** Can be addressed with available methods
- [ ] **Relevant:** Addresses a gap in knowledge or practical need
- [ ] **Feasible:** Resources, time, and ethical considerations allow it
- [ ] **Falsifiable:** Can be proven wrong if incorrect
### Have You Reviewed the Literature?
- [ ] Identified what's already known
- [ ] Found gaps or contradictions to address
- [ ] Learned from methodological successes and failures
- [ ] Identified appropriate outcome measures
- [ ] Determined typical effect sizes in the field
## Hypothesis Development
### Is Your Hypothesis Testable?
- [ ] Makes specific, quantifiable predictions
- [ ] Variables are operationally defined
- [ ] Specifies direction/nature of expected relationships
- [ ] Can be falsified by potential observations
### Types of Hypotheses
- [ ] **Null hypothesis (H₀):** Explicit value or range, including any non-inferiority margin
- [ ] **Alternative hypothesis (H₁):** Explicit competing value, range, or direction
- [ ] **Directional vs. non-directional:** One-tailed vs. two-tailed tests
## Study Design Selection
### What Type of Study is Appropriate?
**Experimental (Intervention) Studies:**
- [ ] **Randomized Controlled Trial (RCT):** Strong intervention-effect design when conduct
and analysis preserve its identifying assumptions
- [ ] **Quasi-experimental:** Non-random assignment but manipulation
- [ ] **Within-subjects:** Same participants in all conditions
- [ ] **Between-subjects:** Different participants per condition
- [ ] **Factorial:** Multiple independent variables
- [ ] **Crossover:** Participants receive multiple interventions sequentially
**Observational Studies:**
- [ ] **Cohort:** Follow groups over time
- [ ] **Case-control:** Compare those with/without outcome
- [ ] **Cross-sectional:** Snapshot at one time point
- [ ] **Ecological:** Population-level data
**Consider:**
- [ ] Can you randomly assign participants?
- [ ] Can you manipulate the independent variable?
- [ ] Is the outcome rare (favor case-control) or common?
- [ ] Do you need to establish temporal sequence?
- [ ] What's feasible given ethical, practical constraints?
## Variables
### Independent Variables (Manipulated/Predictor)
- [ ] Clearly defined and operationalized
- [ ] Appropriate levels/categories chosen
- [ ] Manipulation is sufficient to test hypothesis
- [ ] Manipulation check planned (if applicable)
### Dependent Variables (Outcome/Response)
- [ ] Directly measures the construct of interest
- [ ] Validated and reliable measurement
- [ ] Sensitive enough to detect expected effects
- [ ] Appropriate for statistical analysis planned
- [ ] Primary outcome clearly designated
### Control Variables
- [ ] **Confounding variables identified:**
- Variables that affect both IV and DV
- Alternative explanations for findings
- [ ] **Strategy for control:**
- Randomization
- Matching
- Stratification
- Statistical adjustment
- Restriction (inclusion/exclusion criteria)
- Avoid conditioning on mediators/colliders unless required by the target estimand
### Extraneous Variables
- [ ] Potential sources of noise identified
- [ ] Standardized procedures to minimize
- [ ] Environmental factors controlled
- [ ] Time of day, setting, equipment standardized
## Sampling
### Population Definition
- [ ] **Target population:** Who you want to generalize to
- [ ] **Accessible population:** Who you can actually sample from
- [ ] **Sample:** Who actually participates
- [ ] Difference between these documented
### Sampling Method
- [ ] **Probability sampling (preferred for generalizability):**
- Simple random sampling
- Stratified sampling
- Cluster sampling
- Systematic sampling
- [ ] **Non-probability sampling (common but limits generalizability):**
- Convenience sampling
- Purposive sampling
- Snowball sampling
- Quota sampling
### Sample Size
- [ ] Experimental/sampling unit and independent replicate count stated (animal, donor,
litter, cage, site, culture, etc.); repeated measurements do not increase independent n
- [ ] **A priori power analysis conducted**
- Scientifically meaningful effect and plausible variability, with sensitivity analysis
- Justified power or precision target (.80/.90 are conventions, not universal requirements)
- Significance level (typically .05)
- Statistical test to be used
- [ ] Accounts for expected attrition/dropout
- [ ] Accounts for clustering, repeated measures, and planned multiplicity
- [ ] Sufficient for planned subgroup analyses
- [ ] Practical constraints acknowledged
### Inclusion/Exclusion Criteria
- [ ] Clearly defined and justified
- [ ] Not overly restrictive (limits generalizability)
- [ ] Based on theoretical or practical considerations
- [ ] Ethical considerations addressed
- [ ] Documented and applied consistently
## Blinding and Randomization
### Randomization
- [ ] **What is randomized:**
- Participant assignment to conditions
- Order of conditions (within-subjects)
- Stimuli/items presented
- [ ] **Method of randomization:**
- Computer-generated random numbers
- Random number tables
- Coin flips (for very small studies)
- [ ] **Allocation concealment:**
- Sequence generated before recruitment
- Allocation hidden until after enrollment
- Secure centralized allocation, or sequentially numbered opaque sealed envelopes
with safeguards preventing anticipation or tampering
- [ ] **Stratified randomization:**
- Balance important variables across groups
- Block randomization to ensure equal group sizes
- [ ] **Check randomization:**
- Audit sequence generation and concealment; describe baseline characteristics
- Do not use baseline significance tests as proof of successful/failed randomization
### Blinding
- [ ] Name who was blinded (participants, providers, assessors, analysts), to what, and
how; single/double/triple-blind labels have inconsistent meanings
- [ ] **Blinding feasibility:**
- Is true blinding possible?
- Placebo/sham controls needed?
- Identical appearance of interventions?
- [ ] **Blinding check:**
- Document unblinding and its likely effect on this outcome
- Assignment guesses after treatment can reflect efficacy/adverse effects; do not
interpret them automatically as a test of blinding success
## Control Groups and Conditions
### What Type of Control?
- [ ] **No treatment control:** Natural course of condition
- [ ] **Placebo control:** Inert treatment for comparison
- [ ] **Active control:** Standard treatment comparison
- [ ] **Wait-list control:** Delayed treatment
- [ ] **Attention control:** Matches contact time without active ingredient
### Multiple Conditions
- [ ] Factorial designs for multiple factors
- [ ] Dose-response relationship assessment
- [ ] Mechanism testing with component analyses
## Procedures
### Protocol Development
- [ ] **Detailed, written protocol:**
- Step-by-step procedures
- Scripts for standardized instructions
- Decision rules for handling issues
- Data collection forms
- [ ] Pilot tested before main study
- [ ] Staff trained to criterion
- [ ] Compliance monitoring planned
### Standardization
- [ ] Same instructions for all participants
- [ ] Same equipment and materials
- [ ] Same environment/setting when possible
- [ ] Same assessment timing
- [ ] Deviations from protocol documented
### Data Collection
- [ ] **When collected:**
- Baseline measurements
- Post-intervention
- Follow-up timepoints
- [ ] **Who collects:**
- Trained researchers
- Blinded when possible
- Inter-rater reliability established
- [ ] **How collected:**
- Valid, reliable instruments
- Standardized administration
- Multiple methods if possible (triangulation)
## Measurement
### Validity
- [ ] **Face validity:** Appears to measure construct
- [ ] **Content validity:** Covers all aspects of construct
- [ ] **Criterion validity:** Correlates with gold standard
- Concurrent validity
- Predictive validity
- [ ] **Construct validity:** Measures theoretical construct
- Convergent validity (correlates with related measures)
- Discriminant validity (doesn't correlate with unrelated measures)
### Reliability
- [ ] **Test-retest:** Consistent over time
- [ ] **Internal consistency:** Justify dimensionality and the chosen reliability model;
Cronbach's α alone does not establish a single construct or validity
- [ ] **Inter-rater reliability:** Agreement between raters (Cohen's κ, ICC)
- [ ] **Parallel forms:** Alternative versions consistent
### Measurement Considerations
- [ ] Objective measures preferred when possible
- [ ] Validated instruments used when available
- [ ] Multiple measures of key constructs
- [ ] Sensitivity to change considered
- [ ] Floor/ceiling effects avoided
- [ ] Response formats appropriate
- [ ] Recall periods appropriate
- [ ] Cultural appropriateness considered
## Bias Minimization
### Selection Bias
- [ ] Random sampling when possible
- [ ] Clearly defined eligibility criteria
- [ ] Document who declines and why
- [ ] Minimize self-selection
### Performance Bias
- [ ] Standardized protocols
- [ ] Blinding of providers
- [ ] Monitor protocol adherence
- [ ] Document deviations
### Detection Bias
- [ ] Blinding of outcome assessors
- [ ] Objective measures when possible
- [ ] Standardized assessment procedures
- [ ] Multiple raters with reliability checks
### Attrition Bias
- [ ] Strategies to minimize dropout
- [ ] Track reasons for dropout
- [ ] Compare dropouts to completers
- [ ] Estimand and missing-outcome handling specified; an intention-to-treat label does
not itself remove attrition bias
### Reporting Bias
- [ ] Preregister study and analysis plan
- [ ] Designate primary vs. secondary outcomes
- [ ] Commit to reporting all outcomes
- [ ] Distinguish planned from exploratory analyses
## Data Management
### Data Collection
- [ ] Data collection forms designed and tested
- [ ] REDCap, Qualtrics, or similar platforms
- [ ] Range checks and validation rules
- [ ] Regular backups
- [ ] Secure storage and documented privacy requirements for the institution/jurisdiction
### Data Quality
- [ ] Real-time data validation
- [ ] Regular quality checks
- [ ] Missing data patterns monitored
- [ ] Outliers identified and investigated
- [ ] Protocol deviations documented
### Data Security
- [ ] De-identification procedures
- [ ] Access controls
- [ ] Audit trails
- [ ] Appropriate ethics-board review and applicable privacy-law requirements confirmed
by the responsible institution; this checklist does not certify legal compliance
## Statistical Analysis Planning
### Analysis Plan (Prespecify Before Data Collection)
- [ ] **Primary analysis:**
- Statistical test(s) specified
- Hypothesis clearly stated
- Significance level set (usually α = .05)
- One-tailed or two-tailed
- [ ] **Secondary analyses:**
- Clearly designated as secondary
- Exploratory analyses labeled as such
- [ ] **Multiple comparisons:**
- Adjustment method specified (if needed)
- Hypothesis family covers relevant outcomes, doses, time points, subgroups, and interim
analyses; naming a primary outcome alone does not control all multiplicity
### Assumptions
- [ ] Assumptions of statistical tests identified
- [ ] Plan to check assumptions
- [ ] Prespecified robust or alternative analysis for the same estimand, with assumptions
- [ ] Transformation options considered
### Missing Data
- [ ] Anticipated amount of missingness
- [ ] Missing data mechanism (MCAR, MAR, MNAR)
- [ ] Handling strategy:
- Complete case analysis
- Multiple imputation
- Maximum likelihood
- [ ] Sensitivity analyses planned
### Effect Sizes
- [ ] Appropriate effect size measures identified
- [ ] Will be reported alongside p-values
- [ ] Confidence intervals planned
### Statistical Software
- [ ] Software selected (R, SPSS, Stata, Python, etc.)
- [ ] Version documented
- [ ] Analysis scripts prepared in advance
- [ ] Will be made available (Open Science)
## Ethical Considerations
### Ethical Approval
- [ ] IRB/Ethics committee approval obtained
- [ ] Study registered (ClinicalTrials.gov, etc.) if applicable
- [ ] For medical research involving human participants, consult the current Declaration
of Helsinki (2024 revision) and applicable local ethics requirements before the study
### Informed Consent
- [ ] Voluntary participation
- [ ] Comprehensible explanation
- [ ] Risks and benefits disclosed
- [ ] Right to withdraw without penalty
- [ ] Privacy protections explained
- [ ] Compensation disclosed
### Risk-Benefit Analysis
- [ ] Potential benefits outweigh risks
- [ ] Risks minimized
- [ ] Vulnerable populations protected
- [ ] Data safety monitoring (if high risk)
### Confidentiality
- [ ] Data de-identified
- [ ] Secure storage
- [ ] Limited access
- [ ] Reporting doesn't allow re-identification
## Validity Threats
### Internal Validity (Causation)
- [ ] **History:** External events between measurements
- [ ] **Maturation:** Changes in participants over time
- [ ] **Testing:** Effects of repeated measurement
- [ ] **Instrumentation:** Changes in measurement over time
- [ ] **Regression to mean:** Extreme scores becoming less extreme
- [ ] **Selection:** Groups differ at baseline
- [ ] **Attrition:** Differential dropout
- [ ] **Diffusion:** Control group receives treatment elements
### External Validity (Generalizability)
- [ ] Sample representative of population
- [ ] Setting realistic/natural
- [ ] Treatment typical of real-world implementation
- [ ] Outcome measures ecologically valid
- [ ] Time frame appropriate
### Construct Validity (Measurement)
- [ ] Measures actually tap intended constructs
- [ ] Operations match theoretical definitions
- [ ] No confounding of constructs
- [ ] Adequate coverage of construct
### Statistical Conclusion Validity
- [ ] Adequate statistical power
- [ ] Assumptions met
- [ ] Appropriate tests used
- [ ] Alpha level appropriate
- [ ] Multiple comparisons addressed
## Reporting and Transparency
### Preregistration
- [ ] Study preregistered (OSF, ClinicalTrials.gov, AsPredicted)
- [ ] Hypotheses stated a priori
- [ ] Analysis plan documented
- [ ] Distinguishes confirmatory from exploratory
### Reporting Guidelines
- [ ] **RCT results:** CONSORT 2025 checklist (with applicable design/intervention extensions)
- [ ] **Randomized trial protocols:** SPIRIT 2025
- [ ] **Observational studies:** STROBE checklist
- [ ] **Systematic reviews:** PRISMA 2020 checklist
- [ ] **Diagnostic accuracy studies:** STARD 2015 and applicable extensions (including
STARD-AI for AI diagnostic accuracy studies)
- [ ] **Qualitative interviews/focus groups:** COREQ checklist
- [ ] **Case reports:** CARE guidelines
Reporting checklists improve transparency; they are not risk-of-bias tools or certificates
of valid conduct. Select extensions for the design instead of assuming every older
extension has been revised to the new core checklist.
### Transparency
- [ ] All measures reported
- [ ] All manipulations disclosed
- [ ] Sample size determination explained
- [ ] Exclusion criteria and numbers reported
- [ ] Attrition documented
- [ ] Deviations from protocol noted
- [ ] Conflicts of interest disclosed
### Open Science
- [ ] Data sharing planned (when ethical)
- [ ] Analysis code shared
- [ ] Materials available
- [ ] Preprint posted
- [ ] Open access publication when possible
## Post-Study Considerations
### Data Analysis
- [ ] Follow preregistered plan
- [ ] Clearly label deviations and exploratory analyses
- [ ] Check assumptions
- [ ] Report all outcomes
- [ ] Report effect sizes and CIs, not just p-values
### Interpretation
- [ ] Conclusions supported by data
- [ ] Limitations acknowledged
- [ ] Alternative explanations considered
- [ ] Generalizability discussed
- [ ] Clinical/practical significance addressed
### Dissemination
- [ ] Publish regardless of results (reduce publication bias)
- [ ] Present at conferences
- [ ] Share findings with participants (when appropriate)
- [ ] Communicate to relevant stakeholders
- [ ] Plain language summaries
### Next Steps
- [ ] Replication needed?
- [ ] Follow-up studies identified
- [ ] Mechanism studies planned
- [ ] Clinical applications considered
## Common Pitfalls to Avoid
- [ ] No justified sample-size/precision rationale; absence of a power calculation alone
does not prove that the completed study was underpowered
- [ ] Hypothesis formed after seeing data (HARKing)
- [ ] Lack of blinding with a plausible route to biased care, measurement, or analysis
- [ ] P-hacking (data fishing, optional stopping)
- [ ] Multiple testing without correction → false positives
- [ ] Inadequate control group
- [ ] Confounding not addressed
- [ ] Instruments not validated
- [ ] High attrition not addressed
- [ ] Cherry-picking results to report
- [ ] Causal language from correlational data
- [ ] Ignoring assumptions of statistical tests
- [ ] Not preregistering changes literature bias
- [ ] Conflicts of interest not disclosed
## Final Checklist Before Starting
- [ ] Research question is clear and important
- [ ] Hypothesis is testable and specific
- [ ] Study design is appropriate
- [ ] Sample size is adequate (power analysis)
- [ ] Measures are valid and reliable
- [ ] Confounds are controlled
- [ ] Randomization and blinding implemented
- [ ] Data collection is standardized
- [ ] Analysis plan is prespecified
- [ ] Ethical approval obtained
- [ ] Study is preregistered
- [ ] Resources are sufficient
- [ ] Team is trained
- [ ] Protocol is documented
- [ ] Backup plans exist for problems
## Remember
**Good experimental design is about:**
- Asking clear questions
- Minimizing bias
- Maximizing validity
- Appropriate inference
- Transparency
- Reproducibility
**The best time to think about these issues is before collecting data, not after.**
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |