Skip to content

dimagi-internal/ace / skills/training-llo-guide-eval

training-llo-guide-eval

Unverified

Internal /ace:run step — Grade the LLO guide.

Training LLO Guide — Eval

Grades 6-qa-and-training/training-llo-guide.md. The LLO guide is the operating manual the local-leadership organization uses every day to run the cohort. If it's a wall of theory rather than an action playbook, Phase 9 onboarding stalls.

Fitness axis (out-of-chain anchor). "Covers ≥ 80% of the catalogued daily/weekly loop" is a catalogue tally — a faithful build of a thin PDD passes it by echoing the loop the PDD already enumerated. The fitness dimension is operational_realism, parallel to idea-to-pdd-eval's resource_realism: would these procedures actually run a real cohort — the judgment calls, the messy troubleshooting beyond the catalogued happy-path loop, the "what do I do when two things go wrong at once"? Judge it against a real ops-lead bar, not against the PDD's loop list, and hard-gate a guide that only reproduces the happy path. See skills/_eval-template.md § The out-of-chain fitness requirement.

See skills/_eval-template.md for shared contracts. Provisional rubric — calibration TBD until 3+ shipped LLO guides produce ground truth (see skills/eval-calibration/SKILL.md).

Inputs

Pipeline-only: grades a run's LLO guide, which exists only in a run.

SourceArtifactUsed for
Phase 66-qa-and-training/training-llo-guide.mdartifact under judgment
Phase 11-design/idea-to-pdd.mdanchors operational caps, escalation matrix, archetype
Phase 44-connect/connect-program-setup.mdLLO-facing Connect controls (program/opportunity status, payment approval flow)
Phase 55-ocs/ocs-bot-config.md (if present)bot escalation routing — guide must match how triage actually works

Products

  • 6-qa-and-training/training-llo-guide-eval_verdict.yaml — verdict YAML per _eval-template.md § Verdict YAML contract.

Process

  1. Read inputs from Drive. Read the artifact under judgment from its PUBLISHED file, never your local draft, and re-read it after any republish — see skills/_eval-template.md § Grade the published revision (stock) (an empty published artifact is a BLOCKER, ace#2592). 1b. Run the published-render gate before grading anything: scripts/check-published-doc-render.ts <published Doc URL> --expect-images — see skills/_eval-template.md § Published-render gate (stock). A non-zero exit (raw markdown on the page, or a doc an anonymous reader cannot open) makes this eval's verdict fail, whatever the prose would score (ace#2213). --expect-images because this artifact is illustrated: a guide that renders zero images fails here. Write the Doc URL you checked to the verdict as top-level published_url, and the result as published_render — capture_path keeps naming the run-folder source, so the verdict records both what was composed and what a reader opened.

  2. Build an operational-coverage checklist from the PDD: morning check-in cadence, daily-cap enforcement, payment-approval workflow, escalation triggers (rejected visits, FLW absence, anomaly alerts), end-of-week reporting.

  3. For each checklist entry, check (a) presence in the guide, (b) action-orientation ("if X, do Y"), (c) screenshot grounding when the action involves a UI step.

  4. Run the ops-lead realism test. Adopt the persona of an experienced ops lead who has run cohorts before, and ask: would these procedures actually carry a real cohort, or do they only describe the happy path? Look for the judgment calls, the troubleshooting beyond the catalogued loop (an FLW disputes a rejection; two anomalies fire the same morning; a payment is held and the FLW is angry), the "what if the obvious action doesn't work" branches. Score operational_realism against that real ops bar, NOT against the PDD's enumerated loop.

  5. Apply the rubric and write the verdict YAML.

  6. Surface concerns per _eval-template.md § Auto-surfaced severity rules.

LLM-as-Judge Rubric

Score each dimension 0–10. Weights sum to 1.0.

DimensionWeightAnchored criteria
Operational realism (fitness — out-of-chain)0.20Judged against a real ops-lead bar (parallel to idea-to-pdd-eval's resource_realism): would these procedures actually run a real cohort? Anchor on real operations, NOT on the PDD's enumerated loop. 10 = the guide handles the judgment calls and the messy cases an ops lead actually hits — disputed rejections, simultaneous anomalies, a held payment with an angry FLW, "the obvious action didn't work, now what" — not just the happy-path loop. 6 = the daily loop is sound but the guide assumes everything goes right; troubleshooting is thin. 3 = pure happy-path; the first real-world snag and the LLO is improvising. Hard-gate: a guide that only reproduces the catalogued happy-path loop with no real troubleshooting/judgment-call content scores ≤ 3 → suite verdict fail. PDD silence on the messy cases is a finding here, not a free pass — judge what a guide that actually runs a cohort must contain. Cap at 7 when the guide has no procedure for a PAID record later found false (stop the accrual, what happens to money already paid, the consequence for the worker), because on a pay-per-verified-visit opportunity that is the case review exists to catch. Judge that procedure against training-llo-guide § A paid record found false: a guide that tells the LLO to "reject the visit in Connect" on an auto-verified opportunity is directing them to a 403 and scores as if the procedure were absent (ace#2610). So does a guide that tells the LLO to export a completed-work status sheet, set a row to rejected and re-import it, or to use a "Payment Verification import": no Connect page links that route, so the LLO cannot reach it. The correct procedure holds the record's payment and routes the rejection through the escalation contact (ace#2613).
Operational completeness (conformance)0.25Does the guide cover the full daily/weekly LLO loop catalogued from the PDD (morning check-in, daily caps, escalations, payment approval, end-of-week report)? 10 = ≥ 80% of catalogued ops covered with concrete cadence. 6 = 50–80%; gaps in either escalation or payment. 3 = < 50%; LLO doesn't know what their day looks like. Hard-deduct -3 if escalation triggers are missing entirely.
Action-orientation0.20Each section reads "if X happens, do Y" rather than "the system supports X." 10 = imperative voice with named buttons/screens; LLO can act without re-reading. 6 = mixed; some sections are descriptive. 3 = mostly descriptive prose; LLO would need a synchronous walkthrough.
Screenshot grounding0.15Judge the PUBLISHED Google Doc, not the markdown. First count the images the document actually renders — docs_get on the published doc and count inlineObjects, or count <img in its anonymous HTML export (https://docs.google.com/document/d/<id>/export?format=html). If the guide cites screenshots and renders none, this dimension is 0 and the suite verdict is fail. Then judge grounding: every UI-bound action is SHOWN (or has a clear "Screen: " anchor). 10 = full grounding, every cited frame rendered. 6 = ≥ 70% grounded, no dead links. 3 = < 50% grounded OR ≥ 1 dead screenshot reference. Hard-deduct -3 per dead Drive-ID reference; hard-deduct -3 when a cited filename names a frame the run never captured. The LLO's own screens count most. If every rendered frame is a worker phone screen and none is an LLO-side Connect view, cap this dimension at 6, because the run captures those views in 4-connect/previews/connect-opportunity/ (ace#2610). Do not expect a frame for rejecting a paid record. Connect has no screen for that step: the LLO sends the record to the escalation contact. Do not deduct for its absence. A guide that shows or names a "Payment Verification import" screen is citing a screen that does not exist, and is scored under operational_realism (ace#2613).
Cap & threshold accuracy0.10Numeric caps (daily visits, payment amounts, escalation thresholds) match the PDD verbatim. 10 = no drift. 6 = 1 minor drift. 3 = any contradicted cap. Hard-deduct -5 for any contradicted operational cap. The window counts as a cap. Read start_date/end_date from connect_get_opportunity and run findDayCountDrift (lib/opportunity-window.ts). A duration that contradicts the window is a drift. The commonest case is the job card's "Days to complete" countdown quoted as the window's length (ace#2610).
Escalation pathway clarity0.10When something goes wrong (rejected visit, anomaly, FLW absence), does the LLO know exactly who/where to go? 10 = every tier has a contact, a channel and an expected response time. 6 = contacts named but no channel or response time. 3 = vague ("contact support"). A clearly marked placeholder scores the same as a named contact when the run's inputs, PDD and run_state name nobody in that role, provided the placeholder says who fills it and when (e.g. [to be named by Spark at onboarding — name, email, expected reply time]). Justification: CLAUDE.md § "No inferred backstory" forbids inventing a contact, so a rubric that requires a name makes fabrication the only path to full marks. Penalise instead (a) a contact the guide names that does not trace to the inputs, scored as 3 and surfaced as a BLOCKER for fabricated backstory, and (b) a silent gap where the guide neither names a contact nor marks the placeholder (ace#2610).

Weights sum to 1.0: 0.20 + 0.25 + 0.20 + 0.15 + 0.10 + 0.10 = 1.00.

Hard-deduct rules:

  • Happy-path-only guide with no real troubleshooting/judgment content (operational_realism ≤ 3) → BLOCKER; suite verdict fail. A faithful reproduction of the PDD's loop does not earn a pass.
  • Dead screenshot Drive ID → BLOCKER (cap overall ≤ 5; see screenshot dimension).
  • Guide cites screenshots and the published document renders ZERO → BLOCKER; suite verdict fail. This is the failure the rubric was blind to: a guide cited nine frames by filename and rendered none, and this eval scored it a pass because every word was present (ace#1418). Counting citations cannot detect it. Count what the document renders.
  • LLO guide contradicts a PDD operational cap → BLOCKER.
  • Any single dimension ≤ 3 → suite verdict fail.

Inflation guard. If training-llo-guide self-eval graded itself top-tier and this rubric's overall ≤ 8.0, cap overall at 8.0 and surface a [WARN]. Default no-op until the producer ships a self-eval.

Calibration target (per _eval-template.md § Calibration target boilerplate):

  • Detection rate ≥ 80% of catalogued LLO-guide issues from eval-calibration/known-issues.md § Training LLO guide (catalogue TBD).
  • Inter-run variance ≤ 0.5 across 3 same-model runs.
  • Agreement with self-eval within ±1.5 points.

Provisional until first real run produces ground truth.

Archetypes

ArchetypeRubric tweak
atomic-visitDefault. Coverage checklist centers on per-visit approval flow, daily cap enforcement, single escalation tier.
focus-groupChecklist adds session-scheduling, attendance verification, facilitator-stipend approval. Operational completeness dimension expects FGD-specific entries.
multi-stageChecklist expects per-stage handoff procedures + Stage-Gate-trigger LLO actions.

MCP Tools Used

See skills/_eval-template.md § MCP Tools Used (stock). Plus an optional drive_read_file per referenced screenshot Drive ID — the rubric resolves screenshot links to detect dead references — and docs_get on the published guide to count rendered inlineObjects, which is a required input to the screenshot dimension and cannot be read off the markdown.

Mode Behavior

See skills/_eval-template.md § Mode Behavior (stock).

Dry-Run Behavior

See skills/_eval-template.md § Dry-Run Behavior (stock).

Change Log

DateChangeAuthor
2026-05-09Initial version. 5 dimensions: operational_completeness (0.35), action_orientation (0.25), screenshot_grounding (0.20), cap_threshold_accuracy (0.10), escalation_pathway_clarity (0.10). Provisional rubric — calibration TBD until first real run grades the artifact.ACE team (qa-eval-registry initial buildout)
2026-05-29Added out-of-chain fitness dimension operational_realism (0.20, hard-gate on happy-path-only guides) — judged against a real ops-lead bar (parallel to idea-to-pdd-eval's resource_realism): would the procedures actually run a real cohort (judgment calls, troubleshooting beyond the catalogued loop)? Demoted the catalogue tally operational_completeness 0.35→0.25; action_orientation 0.25→0.20, screenshot_grounding 0.20→0.15. Sum = 1.00. Per docs/superpowers/specs/2026-05-29-eval-fitness-gap.md.ACE team (eval-fitness-gap)
2026-10-05ace#2613: Connect has no UI for the completed-work status import (completed_work_import is linked from no template on commcare-connect @ 046c7fd7). operational_realism now scores a self-service export/re-import procedure, which no LLO can reach, like "reject the visit in Connect": as if the procedure were absent. screenshot_grounding no longer expects a frame for the rejection step.ACE team (ace#2613)
2026-10-02Four changes from ace#2610 (spark/spark-facilitator/20261001-2208, warn 7.6): (1) operational_realism is capped at 7 when there is no procedure for a paid record later found false, and the procedure is judged against the Connect mechanism read from source. (2) screenshot_grounding is capped at 6 when no rendered frame shows an LLO-side screen. (3) In cap_threshold_accuracy, the window counts as a cap, checked with findDayCountDrift. (4) In escalation_pathway_clarity, an honest placeholder scores as a named contact, and a contact that does not trace to the inputs is the deduction, so the rubric no longer rewards inferred backstory.ACE team (ace#2610)
2026-08-14Screenshot grounding now judges the PUBLISHED document's rendered image count (docs_get inlineObjects / anonymous <img> export), not the markdown's citation count. Cites screenshots + renders none → dimension 0 + BLOCKER. Closes the blind spot that scored spark-facilitator/20260813-2126 a pass with 9 filename citations and 0 pictures (ace#1418).ACE team (ace#1418)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 192kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 192kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 192kUpdated Claude CodeCodexVerifiedInstall