DAAF-Contribution-Community/daaf / .claude/skills/creative-verification
creative-verification
VerifiedResearcher-facing analytic triangulation ("creative verification") for stress-testing provisional findings and designing pre-specified challenges before an analysis runs. Turns a researcher's concern or focal claim into competing explanations, bounded discriminating tests, audited diagnostic artifacts, and evidence-calibrated interpretations in a hypothesis/evidence ledger — behavioral, output-level verification replacing line-by-line code review. Covers 12 challenge families organized by Shadish-Cook-Campbell validity type, named sensitivity statistics, pre-committed interpretation rules, anti-fishing guardrails, and interactive diagnostic-artifact QA. Use in Ad Hoc Collaboration to verify, challenge, or stress-test a result, or to design discriminating tests before an analysis runs. Not code-change verification (verify skill / code review) and not final adjudication or re-execution (data-verifier, Reproducibility Verification) — prospective, hypothesis-driven challenge design of analytic claims.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 16.9 KB |
| references/challenge-families.md | markdown | 36.9 KB |
| references/interactive-artifact-qa.md | markdown | 7.9 KB |
| references/interpretation-and-evidence.md | markdown | 12.0 KB |
| references/sensitivity-statistics.md | markdown | 23.8 KB |
references/interactive-artifact-qa.md
View on GitHub# Interactive-Artifact QA
The QA contract for **diagnostic dashboards** — interactive artifacts built to let a
researcher explore a verification question, delivered as Plotly HTML files or
standalone Marimo apps. These artifacts multiply framing risk (Family 8) and are easy
to mistake for evidence, so they carry their own review contract. This file defines
what a diagnostic dashboard must satisfy before it can be trusted as an exploratory
interface, and the boundary rules that keep it from masquerading as a validated result.
**The governing principle.** An interactive artifact is an **exploratory interface, not
self-validating evidence.** Every default view and every interaction state must trace
to audited data produced by captured, immutable scripts. A dashboard that looks
convincing is not thereby correct — the QA below is what makes any state it can display
trustworthy, and even then the dashboard supports *finding* claims, not *proving* them.
The proving is done by the underlying tests in the ledger.
---
## Boundary Rules (from the design record)
These are load-bearing and precede any checklist item:
- **Diagnostic data is Parquet from captured immutable scripts.** Every number a
dashboard displays comes from a Parquet file generated by a file-first script with an
appended execution log — never computed live in undocumented dashboard code, never
read from an ephemeral source. The dashboard reads audited data; it does not create
it.
- **Interactive artifacts are separately planned — never Stage 9 notebooks.** Stage 9
Marimo/Quarto notebooks remain script-and-log archives; they may preview existing
Parquet or show already-created figures but may **not** acquire filters, new analysis,
or dashboard behavior. A diagnostic dashboard is a distinct, separately-planned
artifact with its own generating script and its own QA pass (SKILL.md Guardrails;
session design record).
- **Static report figures stay PNG on the standard path.** The dashboard is for
exploration; anything that goes into a report is a static figure produced by the
normal visualization path, not a screenshot of an interactive state.
- **An interactive artifact is not evidence.** A pattern seen while clicking through a
dashboard becomes evidence only when it is turned into a pre-committed, executed test
in the ledger (`interpretation-and-evidence.md`). For pattern *claims*, the lineup
protocol (below) is the bridge from "I saw it" to "it is unlikely to be chance."
---
## QA Contract
Practitioner dashboard-testing conventions grounded in the pre-authoring survey
(https://bug0.com/knowledge-base/dashboard-testing ;
https://testsigma.com/blog/dashboard-testing/ ; empty-state UX —
https://www.pencilandpaper.io/articles/empty-states). Treat this as best-available
practice (the interactive-artifact-as-evidence literature is thin), not a validated
standard — but every item below must be checked before a diagnostic dashboard is
handed to the researcher.
### 1. Default-State Check
The artifact opens to a **documented, non-cherry-picked** default view. The opening
filter set, sort, and date range are recorded (in the generating script and in the
ledger entry for the artifact) and are *representative*, not the slice that most
flatters the claim. If the default highlights a particular subgroup or window, that
choice is justified in writing.
### 2. Filter Integrity
- **AND/OR semantics are correct and stated.** Combining two filters does what the UI
implies (usually AND across dimensions, OR within a dimension) — verify against a
hand-computed cross-tab.
- **Clearing all filters returns to the documented default**, not to an empty or
arbitrary state.
- **Drill-down preserves active filters.** Clicking into a subgroup keeps the higher-
level filters applied rather than silently resetting them.
### 3. Range / Date Edge Cases
Exercise the boundaries: first and last available period, single-period selection,
month-end and year-end boundaries, and any "last-N-periods vs. current-period"
control. For education-data panels, confirm that boundary years with known coverage
gaps (e.g., a COVID-affected collection year) are handled explicitly rather than
rendered as an ordinary interior point.
### 4. Empty and Suppressed States
- **Empty states say *why* and *what next*.** A view with no data explains the reason
(no rows match, data not collected that year) and points to a next action — never a
blank panel or a misleading zero.
- **Suppressed cells render as "suppressed," never as zero.** Education-data suppression
code `-3` (and missing `-1`, not-applicable `-2`) must display as its semantic label,
not be silently coerced to 0 — coercing suppression to zero is a correctness bug that
fabricates data and can invert a conclusion.
### 5. Number Parity
**Every displayed aggregate is cross-checked against a source query.** Pick each
headline number the dashboard shows and confirm it equals the value from an independent
query against the underlying Parquet (the Family 9 re-derivation habit applied to the
UI). A dashboard whose totals don't tie to the source is not shippable regardless of
how polished it looks.
### 6. Reproducible Export
- **Marimo `.py` stored alongside a static `html-wasm` export.** Marimo is Git-friendly
`.py` and exports to static `html-wasm` that runs fully in-browser with no backend
(https://marimo.io/features/feat-deploy) — store both the `.py` and the static export
next to the generating script so the artifact is auditable and reproducible without a
live server.
- **Plotly HTML is self-contained.** A Plotly HTML diagnostic is a standalone file whose
data is embedded or sits in a committed Parquet the generating script points to.
- **The generating script is captured.** Like every DAAF artifact, the script that
builds the dashboard is a file-first script with an appended execution log; the
dashboard is downstream of an audited script, never hand-assembled.
### 7. Lineup-Protocol Embed (optional, high-value for pattern claims)
For a dashboard that invites a **pattern claim** ("this cluster is real," "this trend
diverges"), optionally embed a lineup-protocol view: the real panel among null-
generated decoys (Wickham, Cook, Hofmann & Buja 2010 —
https://vita.had.co.nz/papers/inference-infovis.pdf). If the researcher can pick the
real panel out of the lineup, the pattern is unlikely to be chance — converting an
eyeballed impression into a calibrated low-α visual test. State the null model used to
generate the decoys; the lineup's validity depends on choosing an appropriate null and
decoy-generating method (Wickham et al. 2010, §4). Bayesian visual predictive checks
carry a related caution (https://arxiv.org/pdf/2503.01509). See Family 8 in `challenge-families.md` for the
generation recipe.
---
## QA Sign-Off
A diagnostic dashboard passes QA when items 1–6 are satisfied (7 where a pattern claim
is present) and each is recorded — the default view documented, filter/edge/empty/
suppressed behavior verified, and at least one number-parity cross-check quoted with
its source query (an *observed* check per `interpretation-and-evidence.md`, not a
recalled one). Record the pass, the artifact paths (`.py` / HTML / static export /
generating script), and the number-parity evidence in the artifact's ledger entry
(`agent_reference/TRIANGULATION_PROTOCOL.md`). An unreviewed dashboard is an
unaudited interface and must not be presented as a verification result.
## References
- Dashboard testing checklists — https://bug0.com/knowledge-base/dashboard-testing ; https://testsigma.com/blog/dashboard-testing/
- Empty-state UX (why + what next) — https://www.pencilandpaper.io/articles/empty-states
- Marimo reproducibility / static `html-wasm` export — https://marimo.io/features/feat-deploy
- Wickham, Cook, Hofmann & Buja (2010), lineup protocol — https://vita.had.co.nz/papers/inference-infovis.pdf
- Bayesian visual predictive checks, caution — https://arxiv.org/pdf/2503.01509Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |