dimagi-internal/ace / skills/synthetic-data-generate-eval
synthetic-data-generate-eval
VerifiedGrade synthetic-data-generate's run summary for record-count health, schema-question coverage, and pre-flight warning handling.
Synthetic Data Generate — Eval
See skills/_eval-template.md for shared verdict / severity / stock-block
contracts. Provisional rubric.
Stage 4 of ACE Phase 7 (Plan B). Grades whether the labs-side synthetic generation actually produced demo-quality data: enough visits, named FLWs with archetype labels round-tripped, deliver-app schema reached, and warnings (payment-units = 0, share-gap, schema-questions = 0) surfaced honestly in the run summary.
Out-of-chain fitness axis (added 2026-05-29 per
docs/superpowers/specs/2026-05-29-eval-fitness-gap.md): the original
five dimensions all graded the run summary against the manifest —
count/field/URL conformance, entirely inside the AI authoring chain. A
manifest-faithful run can still emit records that any domain expert would
clock as garbage at a glance (flat distributions, robotic visit cadence,
no archetype-driven variance). The new data_plausibility dimension reads
the ACTUAL generated records on labs (NOT the manifest) and judges whether
the data would survive a domain expert's glance at the labs dashboard.
This is the dimension that carries teeth — it can fail an otherwise
conformant run.
Status: Provisional.
Inputs
Pipeline-only: grades a run's Phase 7 synthetic dataset, which exists only in a run.
ACE/<opp-name>/runs/<run-id>/7-synthetic/synthetic-data-generate.mdACE/<opp-name>/runs/<run-id>/7-synthetic/synthetic-data-generate_manifest.yamlORsynthetic-narrative-plan.yaml(whichever was the source)ACE/<opp-name>/opp.yaml—synthetic.fixture_record_counts,synthetic.labs_opp_id- The actual generated records on labs (the out-of-chain anchor for
data_plausibility): read a sample viasynthetic_local_record_dump(orsynthetic_local_records_count+mbw_dashboard_v3for the rendered dashboard view) againstsynthetic.labs_opp_id. This is the observed substrate, not the manifest's claim about it.
Rubric
Score each dimension 0–10.
-
Record-count health (weight 0.20).
user_visits >= 50ANDuser_data == flw_personas.lengthfrom the manifest. Hard-deduct -5 ifuser_visits == 0(engine ran but generated nothing — typically timeline misconfig). Hard-deduct -3 ifuser_data != flw_personas.length(named FLWs didn't all round-trip). -
Form schema coverage (weight 0.15).
form_schema_questions > 0means the deliver-app's HQ schema was reached and visitform_jsonhas populated paths. Hard-deduct -5 ifform_schema_questions == 0(visits exist but form fields are sparse — demo can't show FLW data). -
Data plausibility (weight 0.25) — OUT-OF-CHAIN FITNESS dimension. Read a sample of the ACTUAL generated records (via
synthetic_local_record_dumpagainstsynthetic.labs_opp_id) — NOT the manifest — and judge whether the data would fool a domain expert glancing at the labs dashboard. Anchor on observed substrate, not the spec:- Distributions: visit volume per FLW varies in a believable shape (the rockstar out-delivers the struggler; no suspiciously flat or perfectly-uniform counts). Verification rates, timestamps, and geo points spread realistically — not clustered at one value or evenly stepped.
- FLW behavior: the named personas' record streams actually reflect their archetype (struggling FLW shows the gaps/late visits the narrative promised; rockstar's records are dense + clean). A manifest can declare archetypes the records don't embody — that's the failure this dimension catches.
- Anomalies: seeded anomalies are present and look organic in the records (a missing-photo run that's plausibly scattered, not a mechanical every-Nth-row toggle).
- This dimension grades fitness, NOT conformance: it asks "would a real-ops reviewer believe this is field data?" Score 9-10 when the sample is indistinguishable from real FLW data at a glance; 5-7 when it's schema-valid but visibly templated; 0-3 when records are obviously synthetic/garbage (constant fields, nonsense distributions, archetypes not embodied).
- Hard-gate: if records are schema-valid but obviously
synthetic/garbage to a domain expert (this dimension ≤ 3), the
whole eval
verdict: failregardless of the other dimensions' scores. Schema-validity is QA's bar; plausibility is eval's.
-
Warning honesty (weight 0.15). When pre-flight detected
payment_unit_count == 0OR labs-share gap OR schema = 0, the run summary must surface a clear[WARN]banner at the top with the consequence stated. Hard-deduct -3 per silent gap (warnings exist in pre-flight signals but the summary doesn't carry them through). -
Manifest provenance (weight 0.10). The summary names which manifest source was consumed (
synthetic-narrative-plan.yaml,synthetic-data-generate_manifest.yaml, or--manifest <path>). The Stage 2 narrative-plan path is preferred when both exist; the summary must say so explicitly when the Stage-1 default manifest is used despite a narrative-plan being present. -
Operator-actionable next steps (weight 0.15). Summary names the labs URL where data is now visible AND the teardown command (
synthetic_disable(<opp-int-id>)). Generic "see labs" without a concrete URL is a fail.
Hard-deduct triggers
[BLOCKER]if any dimension scores ≤ 3.[BLOCKER](hard-gate,verdict: fail) ifdata_plausibility≤ 3 — records are schema-valid but obviously synthetic/garbage to a domain expert. This gate fires regardless of how high the conformance dimensions score; a faithful-but-unbelievable dataset does not pass.[BLOCKER]ifuser_visits == 0(generation effectively failed).[BLOCKER]if the labs URL in the summary is wrong (e.g., references a connect.dimagi.com pattern instead of labs.connect.dimagi.com).[WARN]ifcompleted_works == 0and the summary doesn't note the payment-unit cause.[WARN]per silent pre-flight gap (warnings dropped from summary).
Verdict shape
Write <7-synthetic-folder>/synthetic-data-generate-eval_verdict.yaml
per lib/verdict-schema.ts:
schema_version: 1
skill: synthetic-data-generate-eval
target: <opp-name>
mode: deep
ran_at: <ISO timestamp>
capture_path: 7-synthetic/synthetic-data-generate.md
overall_score: <weighted mean post-cap>
overall_score_pre_cap: <raw weighted mean>
verdict: pass | warn | fail
dimensions:
record_count_health: { score: <0-10>, weight: 0.20 }
form_schema_coverage: { score: <0-10>, weight: 0.15 }
data_plausibility: { score: <0-10>, weight: 0.25 } # OUT-OF-CHAIN fitness; ≤3 hard-gates the eval
warning_honesty: { score: <0-10>, weight: 0.15 }
manifest_provenance: { score: <0-10>, weight: 0.10 }
operator_next_steps: { score: <0-10>, weight: 0.15 }
# Weights sum: 0.20 + 0.15 + 0.25 + 0.15 + 0.10 + 0.15 = 1.00.
hard_deduct_triggered: [ ... ]
auto_surfaced: [ ... ]
gate:
threshold: 7.0
disposition: approve | iterate | reject
Calibration target
Provisional. Calibrate once 3+ runs have shipped — see
skills/eval-calibration/SKILL.md.
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-05-06 | Initial provisional rubric — Stage 4 of Plan B. | ACE team |
| 2026-05-29 | Add data_plausibility (0.25) out-of-chain fitness dimension — reads the ACTUAL generated records via synthetic_local_record_dump and judges whether they'd fool a domain expert, with a ≤3 hard-gate (verdict: fail) on schema-valid-but-garbage data. Reweight the conformance dims (record_count 0.30→0.20, form_schema 0.25→0.15, warning_honesty 0.20→0.15) to absorb it; weights still sum to 1.00. Closes the all-conformance gap flagged in docs/superpowers/specs/2026-05-29-eval-fitness-gap.md. | ACE team |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |