dimagi-internal/ace / skills/training-flw-guide-eval
training-flw-guide-eval
VerifiedInternal /ace:run step — Grade the FLW guide.
Training FLW Guide — Eval
Grades 6-qa-and-training/training-flw-guide.md. The FLW guide is the
ground-truth onboarding artifact for the field worker. Unlike the FAQ
(reference) or quick-reference (glance), this is the linear walkthrough
they read once before their first visit. If steps are abstract or
screenshots are missing, the first visit fails.
Fitness axis (out-of-chain anchor). "Each step names the visible UI
control" collapses to a presence check — a thin guide can name controls
and still leave a first-timer stranded. The fitness dimension is
step_concreteness, reframed here as a held-out comprehension test:
simulate a naive, first-time worker who has only this guide, and judge
whether they could actually complete the visit end-to-end without asking
anyone. It anchors on real task completion, not on whether the prose
mentions a button, and hard-gates a guide a naive reader couldn't follow.
error_recovery_coverage is weighted higher because the failure modes
(GPS fails, photo blurs, consent refused) are exactly where a naive
first-timer gets stuck. See skills/_eval-template.md § The out-of-chain fitness requirement.
See skills/_eval-template.md for shared contracts. Provisional rubric —
calibration TBD until 3+ shipped FLW guides produce ground truth.
Inputs
Pipeline-only: grades a run's FLW guide, which exists only in a run.
| Source | Artifact | Used for |
|---|---|---|
| Phase 6 | 6-qa-and-training/training-flw-guide.md | artifact under judgment |
| Phase 1 | 1-design/idea-to-pdd.md | anchors archetype + worker-flow expectations + accessibility constraints (language, literacy assumption) |
| Phase 3 | 3-commcare/app-deploy_summary.md | learn/deliver app flow — guide steps must match form-by-form what the FLW sees |
Products
6-qa-and-training/training-flw-guide-eval_verdict.yaml— verdict YAML per_eval-template.md § Verdict YAML contract.
Process
-
Read inputs from Drive. Read the artifact under judgment from its PUBLISHED file, never your local draft, and re-read it after any republish — see
skills/_eval-template.md § Grade the published revision (stock)(an empty published artifact is a BLOCKER, ace#2592). 1b. Run the published-render gate before grading anything:scripts/check-published-doc-render.ts <published Doc URL> --expect-images— seeskills/_eval-template.md § Published-render gate (stock). A non-zero exit (raw markdown on the page, or a doc an anonymous reader cannot open) makes this eval's verdictfail, whatever the prose would score (ace#2213).--expect-imagesbecause this artifact isillustrated: a guide that renders zero images fails here. Write the Doc URL you checked to the verdict as top-levelpublished_url, and the result aspublished_render—capture_pathkeeps naming the run-folder source, so the verdict records both what was composed and what a reader opened. -
Walk the deliver-app flow from
app-deploy_summary.md(form by form, field by field). Build the expected step sequence. -
Run the held-out comprehension test. Adopt the persona of a naive, first-time worker who has only this guide — no prior app exposure, no one to ask. Read it straight through and ask at each step: "do I now know exactly what to tap/enter/do, or have I been left to improvise?" Score
step_concretenesson whether that naive reader reaches a completed visit, NOT merely on whether each step names a control. -
Apply the rubric and write the verdict YAML.
-
Surface concerns per
_eval-template.md § Auto-surfaced severity rules.
LLM-as-Judge Rubric
Score each dimension 0–10. Weights sum to 1.0.
| Dimension | Weight | Anchored criteria |
|---|---|---|
| Step-by-step concreteness (fitness — held-out comprehension test, out-of-chain) | 0.35 | Run the held-out test: a naive, first-time worker with only this guide and no one to ask — do they complete the visit end-to-end, or do they get stranded? Anchor on real task completion, NOT on whether each step happens to name a control (naming a button is necessary but not sufficient — "Tap Submit" is useless if the reader doesn't know what to enter first). 10 = the naive reader completes every form, every decision, every entry, with no point where they'd have to guess or ask. 6 = they get through the main flow but stall at ≥ 1 underspecified step (a field with no guidance on what to enter, a decision with no criterion). 3 = the guide names topics/controls but a first-timer would be improvising for most of the visit. Hard-gate: if the held-out reader cannot complete the visit from the guide alone, score ≤ 3 → suite verdict fail. Hard-deduct -3 if a deliver-app form is skipped entirely. PDD silence on a step is a finding here, not a free pass — judge what a deployable first-visit guide must tell the reader. |
| Error-recovery coverage | 0.20 | What happens when GPS fails, photo blurs, consent is refused, market is closed? These are exactly where the naive first-timer gets stuck, so this is weighted as a primary fitness signal. 10 = explicit recovery path for each Layer A failure mode. 6 = recovery for the common cases (photo retake) but not edges. 3 = no error-recovery section; FLW will be stuck. |
| Screenshot completeness | 0.20 | Judge the PUBLISHED Google Doc, not the markdown. First count the images the document actually renders — docs_get on the published doc and count inlineObjects, or count <img in its anonymous HTML export (https://docs.google.com/document/d/<id>/export?format=html). If that count is zero, this dimension is 0 and the suite verdict is fail, no matter how many references the text carries. Then, and only then, judge coverage: every step that involves a non-trivial UI choice is SHOWN. 10 = full coverage of forms + decision points + error recovery, every cited frame rendered. 6 = main flow shown, edge cases (retake photo, retry GPS) missing. 3 = < 50% of steps shown OR ≥ 1 dead Drive-ID reference. Hard-deduct -3 per dead Drive-ID reference; hard-deduct -3 if the rendered image count is materially below the citation count (references that became text). |
| Language accessibility | 0.15 | Reading level matches the PDD's stated FLW literacy/language constraint. 10 = short sentences, no jargon, names UI controls in the deployed language. 6 = mostly accessible; occasional jargon. 3 = English-only when PDD specifies local language, or developer-jargon throughout. Hard-deduct -5 if guide is in the wrong language entirely. |
| Flow ordering fidelity | 0.10 | Step ordering matches the actual deliver-app form order. 10 = exact match. 6 = ≤ 1 swap. 3 = ≥ 2 ordering errors; FLW reads steps that don't match what they see on-screen. |
Weights sum to 1.0: 0.35 + 0.20 + 0.20 + 0.15 + 0.10 = 1.00.
Hard-deduct rules:
- Held-out reader cannot complete the visit from the guide alone (
step_concreteness≤ 3) → BLOCKER; suite verdictfail. A guide that names controls but leaves a first-timer improvising does not earn a pass. - Dead screenshot Drive ID → BLOCKER (cap overall ≤ 5).
- Published document renders ZERO images → BLOCKER; suite verdict
fail. A step-by-step guide whose steps are not shown is not a step-by-step guide, and this is the failure the rubric was blind to: a guide published 44 screenshot references and 0 pictures, and every content eval — this one included — scored it a pass, because every word was there (ace#1418). Counting references cannot detect it. Count what the document renders. - Guide is in the wrong language vs PDD constraint → BLOCKER.
- Any single dimension ≤ 3 → suite verdict
fail.
Inflation guard. If training-flw-guide self-eval graded itself
top-tier and this rubric's overall ≤ 8.0, cap overall at 8.0 and surface
a [WARN]. Default no-op until the producer ships a self-eval.
Calibration target (per _eval-template.md § Calibration target boilerplate):
- Detection rate ≥ 80% of catalogued FLW-guide issues from
eval-calibration/known-issues.md § Training FLW guide(catalogue TBD). - Inter-run variance ≤ 0.5 across 3 same-model runs.
- Agreement with self-eval within ±1.5 points.
Provisional until first real run produces ground truth.
Archetypes
| Archetype | Rubric tweak |
|---|---|
atomic-visit | Default. Expected step sequence is single-form-flow per visit; error recovery centers on photo/GPS/consent. |
focus-group | Step sequence covers attendance roster, per-domain summary capture, end-of-session sign-off. Concreteness dimension expects FGD facilitation cues. |
multi-stage | Step sequence covers per-stage handoff and Stage-Gate-pending state. Flow-ordering dimension expects stage-aware sequencing. |
MCP Tools Used
See skills/_eval-template.md § MCP Tools Used (stock). Plus optional
drive_read_file per referenced screenshot Drive ID to detect dead
references, and docs_get on the published guide to count rendered
inlineObjects — the rendered-image count is a required input to the
screenshot dimension, and it cannot be read off the markdown.
Mode Behavior
See skills/_eval-template.md § Mode Behavior (stock).
Dry-Run Behavior
See skills/_eval-template.md § Dry-Run Behavior (stock).
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-05-09 | Initial version. 5 dimensions: step_concreteness (0.35), screenshot_completeness (0.25), language_accessibility (0.15), error_recovery_coverage (0.15), flow_ordering_fidelity (0.10). Provisional rubric — calibration TBD until first real run grades the artifact. | ACE team (qa-eval-registry initial buildout) |
| 2026-05-29 | Converted step_concreteness (0.35, unchanged weight) from a "names the visible UI control" presence check into an out-of-chain held-out comprehension test (naive first-timer with only the guide completes the visit), with a hard-gate. Weighted error_recovery_coverage up 0.15→0.20 (where naive readers get stuck); demoted screenshot_completeness 0.25→0.20 to compensate. Sum = 1.00. Per docs/superpowers/specs/2026-05-29-eval-fitness-gap.md. | ACE team (eval-fitness-gap) |
| 2026-08-14 | Screenshot dimension now judges the PUBLISHED document's rendered image count (docs_get inlineObjects / anonymous <img> export), not the markdown's reference count. Zero rendered images → dimension 0 + BLOCKER. Closes the blind spot that scored spark-facilitator/20260813-2126 a pass with 44 references and 0 pictures (ace#1418). | ACE team (ace#1418) |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |