scdenney/open-science-skills / plugin/skills/methods-reporting
methods-reporting
VerifiedAudits a methods section against a 45-item reporting checklist synthesized from the APSA Experimental Section rubric, JARS-Quant, CONSORT, and DA-RT, covering pre-registration, recruitment, randomization and treatments, sample flow and attrition, sample-size justification, three-tier results labeling, conjoint reporting, validity, and open-science infrastructure. Use when checking whether a methods section reports enough, preparing a submission or replication archive, asking what CONSORT, JARS, or DA-RT require, or documenting a deviation from a pre-analysis plan. Writing that plan before data collection goes to pre-registration-writing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.4 KB |
| reference/checklist.md | markdown | 4.9 KB |
| reference/conjoint-reporting.md | markdown | 2.4 KB |
| reference/example-methods-paragraph.md | markdown | 7.2 KB |
reference/checklist.md
View on GitHub# 45-Item Reporting Checklist
This integrated 45-item reporting checklist is a *synthesis* of: the 19-item APSA Experimental Section rubric (Gerber et al. 2014), JARS-Quant (Appelbaum et al. 2018; APA 2020) as summarized in six core pre-registration elements by Lakens (2025), DA-RT principles (Lupia and Elman 2014; Nosek et al. 2015), and conjoint (Hainmueller, Hopkins, and Yamamoto 2014; Bansak et al. 2021) and survey-quality (Stantcheva 2023) best practice. No single authority mandates all 45 items as a unit; items 20--28 and 41--45 extend canonical standards to cover conjoint designs and survey-quality concerns respectively. Treat the checklist as a *recommended* reporting ceiling, not a minimum-mandatory floor. Items are labeled as interrogative checks throughout for consistency.
For a paragraph-level worked example illustrating how items 1--19 can be compactly reported in ~650 words of prose plus named supplementary materials, see [example-methods-paragraph.md](example-methods-paragraph.md).
## Core Reporting (Items 1--19, APSA / Gerber et al. 2014)
- [ ] 1. Eligibility criteria for participants reported?
- [ ] 2. Specific recruitment and conduct dates (including follow-ups) provided?
- [ ] 3. Survey firm identified and recruitment methods described?
- [ ] 4. Response rate and calculation method reported?
- [ ] 5. Use of random assignment explicitly stated?
- [ ] 6. Unit of randomization (individual, group, cluster) identified?
- [ ] 7. Table of baseline means and SDs by experimental group provided?
- [ ] 8. Detailed description of all treatment and control conditions included?
- [ ] 9. Complete treatment materials (scripts, images, etc.) made available?
- [ ] 10. Outcome variable measurement and coding defined?
- [ ] 11. Exact construction of any indices explained?
- [ ] 12. Measurement and coding of all covariates/model variables reported?
- [ ] 13. Number of subjects initially assessed for eligibility reported?
- [ ] 14. Exclusions prior to randomization and reasons provided?
- [ ] 15. Number of subjects assigned to each experimental group stated?
- [ ] 16. Proportion receiving the intervention and reasons for non-receipt reported?
- [ ] 17. Number of subjects per group lacking outcome data reported?
- [ ] 18. Number included in analysis and rationale for exclusions provided?
- [ ] 19. Outcome means/SDs/Ns reported using Intent-to-Treat (ITT) analysis?
## Conjoint-Specific Reporting (Items 20--28, Hainmueller, Hopkins, and Yamamoto 2014; Bansak et al. 2021)
- [ ] 20. Complete attribute table with all levels and reference categories provided?
- [ ] 21. Any randomization constraints (nesting, linking, restrictions) documented and justified?
- [ ] 22. Vignette assembly template and worked examples included (if factorial vignette)?
- [ ] 23. Number of tasks, choice format, and randomization method reported?
- [ ] 24. Primary and secondary DVs clearly distinguished with exact question wording?
- [ ] 25. Post-block items (attribution, manipulation checks) documented with exact wording?
- [ ] 26. Effective sample size (profile evaluations) and per-cell Ns reported?
- [ ] 27. AMCE estimation method, clustering, and marginal means computation specified?
- [ ] 28. All pre-registered interaction models and planned figures listed?
## Design Transparency (Items 29--40)
- [ ] 29. Hypotheses stated prior to research design, not post hoc (Gerber et al. 2014)?
- [ ] 30. Subject incentives described (Gerber et al. 2014)?
- [ ] 31. Blinding status (single, double, or none) reported (Gerber et al. 2014)?
- [ ] 32. Mode of treatment delivery (text, audio, video, in-person) described (Gerber et al. 2014)?
- [ ] 33. Randomization software/tool and assignment sequence details provided (Gerber et al. 2014)?
- [ ] 34. Manipulation checks documented with placement, wording, and results (Druckman 2022)?
- [ ] 35. Sample size justification type identified and reported (Lakens 2025)?
- [ ] 36. SESOI specified and justified for all hypothesis tests, or rationale given for its absence (Lakens 2025)?
- [ ] 37. Pre-registration ID and repository stated (Druckman 2022)?
- [ ] 38. Deviations from pre-analysis plan documented with severity and validity impact (Lakens 2025)?
- [ ] 39. Results labeled as primary, secondary, or exploratory (Appelbaum et al. 2018 JARS-Quant three-group distinction; Lakens 2025)?
- [ ] 40. Data sharing plan, reproducible analysis code, and open materials platform specified (Christensen et al. 2019)?
## Survey Design Quality (Items 41--45, Stantcheva 2023)
- [ ] 41. Was a soft-launch conducted before full deployment, with technical issues documented?
- [ ] 42. Are raw (unadjusted) results reported alongside any corrected/reweighted results?
- [ ] 43. Are benchmark comparisons to existing representative surveys reported to validate sample quality?
- [ ] 44. Have question wording best practices been followed (item-specific scales, no agree-disagree, randomized option order)?
- [ ] 45. Is the three-stage survey error pipeline (coverage, sampling, nonresponse) addressed?
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |