Skip to content

scdenney/open-science-skills / plugin/skills/methods-reporting

methods-reporting

Verified

Audits a methods section against a 45-item reporting checklist synthesized from the APSA Experimental Section rubric, JARS-Quant, CONSORT, and DA-RT, covering pre-registration, recruitment, randomization and treatments, sample flow and attrition, sample-size justification, three-tier results labeling, conjoint reporting, validity, and open-science infrastructure. Use when checking whether a methods section reports enough, preparing a submission or replication archive, asking what CONSORT, JARS, or DA-RT require, or documenting a deviation from a pre-analysis plan. Writing that plan before data collection goes to pre-registration-writing.

FileKindSize
SKILL.mdskill md19.4 KB
reference/checklist.mdmarkdown4.9 KB
reference/conjoint-reporting.mdmarkdown2.4 KB
reference/example-methods-paragraph.mdmarkdown7.2 KB

reference/conjoint-reporting.md

View on GitHub
# Conjoint-Specific Reporting

This section applies when the experiment uses a conjoint or factorial vignette design. The primary methodological references for these items are Hainmueller, Hopkins, and Yamamoto (2014) -- the foundational AMCE identification and estimation paper -- and Bansak, Hainmueller, Hopkins, and Yamamoto (2021, *Advances in Experimental Political Science*), which consolidates design and reporting recommendations.

- **Attribute Table:** Report a complete table listing all attributes, their levels, and the reference category for each. If levels are country-specific (localized), provide both the generic attribute definition and the country-specific operationalizations (Hainmueller, Hopkins, and Yamamoto 2014).
- **Randomization Constraints:** If any attributes are nested, linked, or restricted (not fully independently randomized), document: (a) which attributes are constrained, (b) the justification for the constraint, and (c) which attributes remain fully independent. This is critical because constrained randomization changes the interpretation of AMCEs (Hainmueller, Hopkins, and Yamamoto 2014; Bansak et al. 2021).
- **Vignette Assembly:** If using factorial vignettes (paragraph-form rather than attribute tables), provide: (a) the template showing how attributes are assembled into a coherent vignette, (b) worked examples of assembled vignettes for multiple attribute combinations, and (c) all country-specific text for each attribute level.
- **Task Structure:** Report: the number of paired-choice tasks per respondent, whether tasks involve forced choice or rating or both, attribute order randomization method, and profile position randomization (Hainmueller, Hopkins, and Yamamoto 2014; Bansak et al. 2021).
- **DV Specification:** Clearly distinguish primary and secondary dependent variables. State the exact question wording for each. If both forced choice and rating are used, specify which is primary (for AMCE estimation) and which is secondary (for robustness).
- **Post-Block Items:** Document any post-block measures (attribution items, manipulation checks) with exact wording and response options. Clearly label these as non-confirmatory if they are not primary outcomes.
- **Effective Sample Size:** Report the effective number of profile evaluations (Respondents × Tasks × Profiles per task) and the number of observations per cell in the attribute space (Hainmueller, Hopkins, and Yamamoto 2014).
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall