Skip to content

dimagi-internal/ace / skills/demo-fidelity-check

demo-fidelity-check

Verified

Pass/fail check that a cloned synthetic dataset reproduces the real source's statistical shape before a funder sees it. Needs the clone and its source opp.

Demo Fidelity Check

Runs only in the clone provider path of demo-data-setup. A cloned demo claims to mirror a real program — so before it's shown to a funder, verify the synthetic fixtures actually reproduce the source's distributions. A low-fidelity clone shown as "this is the real program" is the worst failure mode this pipeline has; this gate prevents it.

See skills/_qa-template.md for the shared QA contract (verdict YAML, auto-fix protocol, static-vs-LLM rules).

Inputs

ParameterUsed forResolver
bundle_dirthe profile bundle under fidelity check (from synthetic_profile_from_prod / synthetic_clone_profile)explicit arg; default: the demo-data-setup (clone) handoff
providermust be clone; anything else skipsexplicit arg; default: run_state …products.synthetic.source

Given bundle_dir, this runs on any cloned dataset, with or without a demo run.

Products

  • <demo-run>/7-synthetic/demo-fidelity-check_result.yaml — QA result per lib/qa-types.ts

Process

  1. Provider guard. If source.provider != clone, emit verdict: skipped with detail "fidelity check is clone-only" and stop. (denovo/ace-run invent their own shape; there is no real source to compare against.)
  2. Run the report. mcp__connect-labs__synthetic_fidelity_report(bundle_dir) — returns per-field mean/std/TVD deltas and a correlation Frobenius distance between the generated fixtures and the manifest they came from.
  3. Score against thresholds (below). Any BLOCKER breach → fail; else pass.

Checks

#idtypedescriptionauto-fix on fail
1report_ranstaticsynthetic_fidelity_report returned a report for the bundlere-check bundle_dir path; re-profile if the bundle is missing
2field_distributions_closestaticevery profiled field's TVD ≤ 0.15 and |mean delta| within tolerance (tune per rubric)regenerate with mirror=true (or pin form_json_paths) so the clone tracks the source shape
3correlations_preservedstaticcorrelation Frobenius distance ≤ threshold — the clone preserves cross-field structure, not just marginalsuse mirror=true (transplant pool) — marginal-only mode loses per-entity structure
4trajectories_presentstaticfor a longitudinal demo (e.g. per-child MUAC recovery), each entity has its multi-visit series (mirror pool populated)re-profile with mirror=true; marginal mode collapses trajectories

Thresholds are provisional pending calibration against ≥3 real clones (mark the rubric provisional per docs/eval-calibration-learnings.md). On fail, the safe fallback is to author the demo denovo rather than show a low-fidelity clone.

slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall