Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/creative-verification

creative-verification

Verified

Researcher-facing analytic triangulation ("creative verification") for stress-testing provisional findings and designing pre-specified challenges before an analysis runs. Turns a researcher's concern or focal claim into competing explanations, bounded discriminating tests, audited diagnostic artifacts, and evidence-calibrated interpretations in a hypothesis/evidence ledger — behavioral, output-level verification replacing line-by-line code review. Covers 12 challenge families organized by Shadish-Cook-Campbell validity type, named sensitivity statistics, pre-committed interpretation rules, anti-fishing guardrails, and interactive diagnostic-artifact QA. Use in Ad Hoc Collaboration to verify, challenge, or stress-test a result, or to design discriminating tests before an analysis runs. Not code-change verification (verify skill / code review) and not final adjudication or re-execution (data-verifier, Reproducibility Verification) — prospective, hypothesis-driven challenge design of analytic claims.

references/interactive-artifact-qa.md

View on GitHub
# Interactive-Artifact QA

The QA contract for **diagnostic dashboards** — interactive artifacts built to let a
researcher explore a verification question, delivered as Plotly HTML files or
standalone Marimo apps. These artifacts multiply framing risk (Family 8) and are easy
to mistake for evidence, so they carry their own review contract. This file defines
what a diagnostic dashboard must satisfy before it can be trusted as an exploratory
interface, and the boundary rules that keep it from masquerading as a validated result.

**The governing principle.** An interactive artifact is an **exploratory interface, not
self-validating evidence.** Every default view and every interaction state must trace
to audited data produced by captured, immutable scripts. A dashboard that looks
convincing is not thereby correct — the QA below is what makes any state it can display
trustworthy, and even then the dashboard supports *finding* claims, not *proving* them.
The proving is done by the underlying tests in the ledger.

---

## Boundary Rules (from the design record)

These are load-bearing and precede any checklist item:

- **Diagnostic data is Parquet from captured immutable scripts.** Every number a
  dashboard displays comes from a Parquet file generated by a file-first script with an
  appended execution log — never computed live in undocumented dashboard code, never
  read from an ephemeral source. The dashboard reads audited data; it does not create
  it.
- **Interactive artifacts are separately planned — never Stage 9 notebooks.** Stage 9
  Marimo/Quarto notebooks remain script-and-log archives; they may preview existing
  Parquet or show already-created figures but may **not** acquire filters, new analysis,
  or dashboard behavior. A diagnostic dashboard is a distinct, separately-planned
  artifact with its own generating script and its own QA pass (SKILL.md Guardrails;
  session design record).
- **Static report figures stay PNG on the standard path.** The dashboard is for
  exploration; anything that goes into a report is a static figure produced by the
  normal visualization path, not a screenshot of an interactive state.
- **An interactive artifact is not evidence.** A pattern seen while clicking through a
  dashboard becomes evidence only when it is turned into a pre-committed, executed test
  in the ledger (`interpretation-and-evidence.md`). For pattern *claims*, the lineup
  protocol (below) is the bridge from "I saw it" to "it is unlikely to be chance."

---

## QA Contract

Practitioner dashboard-testing conventions grounded in the pre-authoring survey
(https://bug0.com/knowledge-base/dashboard-testing ;
https://testsigma.com/blog/dashboard-testing/ ; empty-state UX —
https://www.pencilandpaper.io/articles/empty-states). Treat this as best-available
practice (the interactive-artifact-as-evidence literature is thin), not a validated
standard — but every item below must be checked before a diagnostic dashboard is
handed to the researcher.

### 1. Default-State Check

The artifact opens to a **documented, non-cherry-picked** default view. The opening
filter set, sort, and date range are recorded (in the generating script and in the
ledger entry for the artifact) and are *representative*, not the slice that most
flatters the claim. If the default highlights a particular subgroup or window, that
choice is justified in writing.

### 2. Filter Integrity

- **AND/OR semantics are correct and stated.** Combining two filters does what the UI
  implies (usually AND across dimensions, OR within a dimension) — verify against a
  hand-computed cross-tab.
- **Clearing all filters returns to the documented default**, not to an empty or
  arbitrary state.
- **Drill-down preserves active filters.** Clicking into a subgroup keeps the higher-
  level filters applied rather than silently resetting them.

### 3. Range / Date Edge Cases

Exercise the boundaries: first and last available period, single-period selection,
month-end and year-end boundaries, and any "last-N-periods vs. current-period"
control. For education-data panels, confirm that boundary years with known coverage
gaps (e.g., a COVID-affected collection year) are handled explicitly rather than
rendered as an ordinary interior point.

### 4. Empty and Suppressed States

- **Empty states say *why* and *what next*.** A view with no data explains the reason
  (no rows match, data not collected that year) and points to a next action — never a
  blank panel or a misleading zero.
- **Suppressed cells render as "suppressed," never as zero.** Education-data suppression
  code `-3` (and missing `-1`, not-applicable `-2`) must display as its semantic label,
  not be silently coerced to 0 — coercing suppression to zero is a correctness bug that
  fabricates data and can invert a conclusion.

### 5. Number Parity

**Every displayed aggregate is cross-checked against a source query.** Pick each
headline number the dashboard shows and confirm it equals the value from an independent
query against the underlying Parquet (the Family 9 re-derivation habit applied to the
UI). A dashboard whose totals don't tie to the source is not shippable regardless of
how polished it looks.

### 6. Reproducible Export

- **Marimo `.py` stored alongside a static `html-wasm` export.** Marimo is Git-friendly
  `.py` and exports to static `html-wasm` that runs fully in-browser with no backend
  (https://marimo.io/features/feat-deploy) — store both the `.py` and the static export
  next to the generating script so the artifact is auditable and reproducible without a
  live server.
- **Plotly HTML is self-contained.** A Plotly HTML diagnostic is a standalone file whose
  data is embedded or sits in a committed Parquet the generating script points to.
- **The generating script is captured.** Like every DAAF artifact, the script that
  builds the dashboard is a file-first script with an appended execution log; the
  dashboard is downstream of an audited script, never hand-assembled.

### 7. Lineup-Protocol Embed (optional, high-value for pattern claims)

For a dashboard that invites a **pattern claim** ("this cluster is real," "this trend
diverges"), optionally embed a lineup-protocol view: the real panel among null-
generated decoys (Wickham, Cook, Hofmann & Buja 2010 —
https://vita.had.co.nz/papers/inference-infovis.pdf). If the researcher can pick the
real panel out of the lineup, the pattern is unlikely to be chance — converting an
eyeballed impression into a calibrated low-α visual test. State the null model used to
generate the decoys; the lineup's validity depends on choosing an appropriate null and
decoy-generating method (Wickham et al. 2010, §4). Bayesian visual predictive checks
carry a related caution (https://arxiv.org/pdf/2503.01509). See Family 8 in `challenge-families.md` for the
generation recipe.

---

## QA Sign-Off

A diagnostic dashboard passes QA when items 1–6 are satisfied (7 where a pattern claim
is present) and each is recorded — the default view documented, filter/edge/empty/
suppressed behavior verified, and at least one number-parity cross-check quoted with
its source query (an *observed* check per `interpretation-and-evidence.md`, not a
recalled one). Record the pass, the artifact paths (`.py` / HTML / static export /
generating script), and the number-parity evidence in the artifact's ledger entry
(`agent_reference/TRIANGULATION_PROTOCOL.md`). An unreviewed dashboard is an
unaudited interface and must not be presented as a verification result.

## References

- Dashboard testing checklists — https://bug0.com/knowledge-base/dashboard-testing ; https://testsigma.com/blog/dashboard-testing/
- Empty-state UX (why + what next) — https://www.pencilandpaper.io/articles/empty-states
- Marimo reproducibility / static `html-wasm` export — https://marimo.io/features/feat-deploy
- Wickham, Cook, Hofmann & Buja (2010), lineup protocol — https://vita.had.co.nz/papers/inference-infovis.pdf
- Bayesian visual predictive checks, caution — https://arxiv.org/pdf/2503.01509
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall