dimagi-internal/ace / skills/training-llo-guide-eval
training-llo-guide-eval
UnverifiedInternal /ace:run step — Grade the LLO guide.
Training LLO Guide — Eval
Grades 6-qa-and-training/training-llo-guide.md. The LLO guide is the
operating manual the local-leadership organization uses every day to run
the cohort. If it's a wall of theory rather than an action playbook,
Phase 9 onboarding stalls.
Fitness axis (out-of-chain anchor). "Covers ≥ 80% of the catalogued
daily/weekly loop" is a catalogue tally — a faithful build of a thin PDD
passes it by echoing the loop the PDD already enumerated. The fitness
dimension is operational_realism, parallel to idea-to-pdd-eval's
resource_realism: would these procedures actually run a real cohort —
the judgment calls, the messy troubleshooting beyond the catalogued
happy-path loop, the "what do I do when two things go wrong at once"?
Judge it against a real ops-lead bar, not against the PDD's loop list, and
hard-gate a guide that only reproduces the happy path. See
skills/_eval-template.md § The out-of-chain fitness requirement.
See skills/_eval-template.md for shared contracts. Provisional rubric —
calibration TBD until 3+ shipped LLO guides produce ground truth (see
skills/eval-calibration/SKILL.md).
Inputs
Pipeline-only: grades a run's LLO guide, which exists only in a run.
| Source | Artifact | Used for |
|---|---|---|
| Phase 6 | 6-qa-and-training/training-llo-guide.md | artifact under judgment |
| Phase 1 | 1-design/idea-to-pdd.md | anchors operational caps, escalation matrix, archetype |
| Phase 4 | 4-connect/connect-program-setup.md | LLO-facing Connect controls (program/opportunity status, payment approval flow) |
| Phase 5 | 5-ocs/ocs-bot-config.md (if present) | bot escalation routing — guide must match how triage actually works |
Products
6-qa-and-training/training-llo-guide-eval_verdict.yaml— verdict YAML per_eval-template.md § Verdict YAML contract.
Process
-
Read inputs from Drive. Read the artifact under judgment from its PUBLISHED file, never your local draft, and re-read it after any republish — see
skills/_eval-template.md § Grade the published revision (stock)(an empty published artifact is a BLOCKER, ace#2592). 1b. Run the published-render gate before grading anything:scripts/check-published-doc-render.ts <published Doc URL> --expect-images— seeskills/_eval-template.md § Published-render gate (stock). A non-zero exit (raw markdown on the page, or a doc an anonymous reader cannot open) makes this eval's verdictfail, whatever the prose would score (ace#2213).--expect-imagesbecause this artifact isillustrated: a guide that renders zero images fails here. Write the Doc URL you checked to the verdict as top-levelpublished_url, and the result aspublished_render—capture_pathkeeps naming the run-folder source, so the verdict records both what was composed and what a reader opened. -
Build an operational-coverage checklist from the PDD: morning check-in cadence, daily-cap enforcement, payment-approval workflow, escalation triggers (rejected visits, FLW absence, anomaly alerts), end-of-week reporting.
-
For each checklist entry, check (a) presence in the guide, (b) action-orientation ("if X, do Y"), (c) screenshot grounding when the action involves a UI step.
-
Run the ops-lead realism test. Adopt the persona of an experienced ops lead who has run cohorts before, and ask: would these procedures actually carry a real cohort, or do they only describe the happy path? Look for the judgment calls, the troubleshooting beyond the catalogued loop (an FLW disputes a rejection; two anomalies fire the same morning; a payment is held and the FLW is angry), the "what if the obvious action doesn't work" branches. Score
operational_realismagainst that real ops bar, NOT against the PDD's enumerated loop. -
Apply the rubric and write the verdict YAML.
-
Surface concerns per
_eval-template.md § Auto-surfaced severity rules.
LLM-as-Judge Rubric
Score each dimension 0–10. Weights sum to 1.0.
| Dimension | Weight | Anchored criteria |
|---|---|---|
| Operational realism (fitness — out-of-chain) | 0.20 | Judged against a real ops-lead bar (parallel to idea-to-pdd-eval's resource_realism): would these procedures actually run a real cohort? Anchor on real operations, NOT on the PDD's enumerated loop. 10 = the guide handles the judgment calls and the messy cases an ops lead actually hits — disputed rejections, simultaneous anomalies, a held payment with an angry FLW, "the obvious action didn't work, now what" — not just the happy-path loop. 6 = the daily loop is sound but the guide assumes everything goes right; troubleshooting is thin. 3 = pure happy-path; the first real-world snag and the LLO is improvising. Hard-gate: a guide that only reproduces the catalogued happy-path loop with no real troubleshooting/judgment-call content scores ≤ 3 → suite verdict fail. PDD silence on the messy cases is a finding here, not a free pass — judge what a guide that actually runs a cohort must contain. Cap at 7 when the guide has no procedure for a PAID record later found false (stop the accrual, what happens to money already paid, the consequence for the worker), because on a pay-per-verified-visit opportunity that is the case review exists to catch. Judge that procedure against training-llo-guide § A paid record found false: a guide that tells the LLO to "reject the visit in Connect" on an auto-verified opportunity is directing them to a 403 and scores as if the procedure were absent (ace#2610). So does a guide that tells the LLO to export a completed-work status sheet, set a row to rejected and re-import it, or to use a "Payment Verification import": no Connect page links that route, so the LLO cannot reach it. The correct procedure holds the record's payment and routes the rejection through the escalation contact (ace#2613). |
| Operational completeness (conformance) | 0.25 | Does the guide cover the full daily/weekly LLO loop catalogued from the PDD (morning check-in, daily caps, escalations, payment approval, end-of-week report)? 10 = ≥ 80% of catalogued ops covered with concrete cadence. 6 = 50–80%; gaps in either escalation or payment. 3 = < 50%; LLO doesn't know what their day looks like. Hard-deduct -3 if escalation triggers are missing entirely. |
| Action-orientation | 0.20 | Each section reads "if X happens, do Y" rather than "the system supports X." 10 = imperative voice with named buttons/screens; LLO can act without re-reading. 6 = mixed; some sections are descriptive. 3 = mostly descriptive prose; LLO would need a synchronous walkthrough. |
| Screenshot grounding | 0.15 | Judge the PUBLISHED Google Doc, not the markdown. First count the images the document actually renders — docs_get on the published doc and count inlineObjects, or count <img in its anonymous HTML export (https://docs.google.com/document/d/<id>/export?format=html). If the guide cites screenshots and renders none, this dimension is 0 and the suite verdict is fail. Then judge grounding: every UI-bound action is SHOWN (or has a clear "Screen: " anchor). 10 = full grounding, every cited frame rendered. 6 = ≥ 70% grounded, no dead links. 3 = < 50% grounded OR ≥ 1 dead screenshot reference. Hard-deduct -3 per dead Drive-ID reference; hard-deduct -3 when a cited filename names a frame the run never captured. The LLO's own screens count most. If every rendered frame is a worker phone screen and none is an LLO-side Connect view, cap this dimension at 6, because the run captures those views in 4-connect/previews/connect-opportunity/ (ace#2610). Do not expect a frame for rejecting a paid record. Connect has no screen for that step: the LLO sends the record to the escalation contact. Do not deduct for its absence. A guide that shows or names a "Payment Verification import" screen is citing a screen that does not exist, and is scored under operational_realism (ace#2613). |
| Cap & threshold accuracy | 0.10 | Numeric caps (daily visits, payment amounts, escalation thresholds) match the PDD verbatim. 10 = no drift. 6 = 1 minor drift. 3 = any contradicted cap. Hard-deduct -5 for any contradicted operational cap. The window counts as a cap. Read start_date/end_date from connect_get_opportunity and run findDayCountDrift (lib/opportunity-window.ts). A duration that contradicts the window is a drift. The commonest case is the job card's "Days to complete" countdown quoted as the window's length (ace#2610). |
| Escalation pathway clarity | 0.10 | When something goes wrong (rejected visit, anomaly, FLW absence), does the LLO know exactly who/where to go? 10 = every tier has a contact, a channel and an expected response time. 6 = contacts named but no channel or response time. 3 = vague ("contact support"). A clearly marked placeholder scores the same as a named contact when the run's inputs, PDD and run_state name nobody in that role, provided the placeholder says who fills it and when (e.g. [to be named by Spark at onboarding — name, email, expected reply time]). Justification: CLAUDE.md § "No inferred backstory" forbids inventing a contact, so a rubric that requires a name makes fabrication the only path to full marks. Penalise instead (a) a contact the guide names that does not trace to the inputs, scored as 3 and surfaced as a BLOCKER for fabricated backstory, and (b) a silent gap where the guide neither names a contact nor marks the placeholder (ace#2610). |
Weights sum to 1.0: 0.20 + 0.25 + 0.20 + 0.15 + 0.10 + 0.10 = 1.00.
Hard-deduct rules:
- Happy-path-only guide with no real troubleshooting/judgment content (
operational_realism≤ 3) → BLOCKER; suite verdictfail. A faithful reproduction of the PDD's loop does not earn a pass. - Dead screenshot Drive ID → BLOCKER (cap overall ≤ 5; see screenshot dimension).
- Guide cites screenshots and the published document renders ZERO → BLOCKER; suite verdict
fail. This is the failure the rubric was blind to: a guide cited nine frames by filename and rendered none, and this eval scored it a pass because every word was present (ace#1418). Counting citations cannot detect it. Count what the document renders. - LLO guide contradicts a PDD operational cap → BLOCKER.
- Any single dimension ≤ 3 → suite verdict
fail.
Inflation guard. If training-llo-guide self-eval graded itself
top-tier and this rubric's overall ≤ 8.0, cap overall at 8.0 and surface
a [WARN]. Default no-op until the producer ships a self-eval.
Calibration target (per _eval-template.md § Calibration target boilerplate):
- Detection rate ≥ 80% of catalogued LLO-guide issues from
eval-calibration/known-issues.md § Training LLO guide(catalogue TBD). - Inter-run variance ≤ 0.5 across 3 same-model runs.
- Agreement with self-eval within ±1.5 points.
Provisional until first real run produces ground truth.
Archetypes
| Archetype | Rubric tweak |
|---|---|
atomic-visit | Default. Coverage checklist centers on per-visit approval flow, daily cap enforcement, single escalation tier. |
focus-group | Checklist adds session-scheduling, attendance verification, facilitator-stipend approval. Operational completeness dimension expects FGD-specific entries. |
multi-stage | Checklist expects per-stage handoff procedures + Stage-Gate-trigger LLO actions. |
MCP Tools Used
See skills/_eval-template.md § MCP Tools Used (stock). Plus an
optional drive_read_file per referenced screenshot Drive ID — the
rubric resolves screenshot links to detect dead references — and
docs_get on the published guide to count rendered inlineObjects,
which is a required input to the screenshot dimension and cannot be read
off the markdown.
Mode Behavior
See skills/_eval-template.md § Mode Behavior (stock).
Dry-Run Behavior
See skills/_eval-template.md § Dry-Run Behavior (stock).
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-05-09 | Initial version. 5 dimensions: operational_completeness (0.35), action_orientation (0.25), screenshot_grounding (0.20), cap_threshold_accuracy (0.10), escalation_pathway_clarity (0.10). Provisional rubric — calibration TBD until first real run grades the artifact. | ACE team (qa-eval-registry initial buildout) |
| 2026-05-29 | Added out-of-chain fitness dimension operational_realism (0.20, hard-gate on happy-path-only guides) — judged against a real ops-lead bar (parallel to idea-to-pdd-eval's resource_realism): would the procedures actually run a real cohort (judgment calls, troubleshooting beyond the catalogued loop)? Demoted the catalogue tally operational_completeness 0.35→0.25; action_orientation 0.25→0.20, screenshot_grounding 0.20→0.15. Sum = 1.00. Per docs/superpowers/specs/2026-05-29-eval-fitness-gap.md. | ACE team (eval-fitness-gap) |
| 2026-10-05 | ace#2613: Connect has no UI for the completed-work status import (completed_work_import is linked from no template on commcare-connect @ 046c7fd7). operational_realism now scores a self-service export/re-import procedure, which no LLO can reach, like "reject the visit in Connect": as if the procedure were absent. screenshot_grounding no longer expects a frame for the rejection step. | ACE team (ace#2613) |
| 2026-10-02 | Four changes from ace#2610 (spark/spark-facilitator/20261001-2208, warn 7.6): (1) operational_realism is capped at 7 when there is no procedure for a paid record later found false, and the procedure is judged against the Connect mechanism read from source. (2) screenshot_grounding is capped at 6 when no rendered frame shows an LLO-side screen. (3) In cap_threshold_accuracy, the window counts as a cap, checked with findDayCountDrift. (4) In escalation_pathway_clarity, an honest placeholder scores as a named contact, and a contact that does not trace to the inputs is the deduction, so the rubric no longer rewards inferred backstory. | ACE team (ace#2610) |
| 2026-08-14 | Screenshot grounding now judges the PUBLISHED document's rendered image count (docs_get inlineObjects / anonymous <img> export), not the markdown's citation count. Cites screenshots + renders none → dimension 0 + BLOCKER. Closes the blind spot that scored spark-facilitator/20260813-2126 a pass with 9 filename citations and 0 pictures (ace#1418). | ACE team (ace#1418) |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |