dimagi-internal/ace / skills/connect-opp-setup-eval
connect-opp-setup-eval
UnverifiedInternal /ace:run step — Grade the run's Connect opportunity setup against its PDD.
Connect Opp Setup Eval
Worker role (multi-role runs): when the dispatch carries a worker_role: {key, label} block, scope this run to that role's apps and its part of the PDD, and read and write every 3-commcare/ (or 4-connect/) file at the same name under 3-commcare/roles/<key>/ (4-connect/roles/<key>/). No block = a single-role run, unchanged. Contract: skills/_worker-roles.md.
connect-opp-setup is the Phase 4 skill that creates the Connect
Opportunity that hangs off the Program — verification flags, payment
units, deliver-unit links, active window. Where connect-program-setup
grades the program-side translation, this rubric grades the
opportunity-side. Together they cover the full Phase 4 Connect-side
build.
Five of the six dimensions triangulate PDD ↔ Connect-config ↔
Deliver-summary — that's fidelity to the AI authoring chain, not
fitness of the opportunity to deploy. Per
skills/_eval-template.md § The out-of-chain fitness requirement, a
rubric whose every dimension is satisfiable by a config that matches the
PDD but would wrongly reject legitimate FLW deliveries (or pay a rate no
real FLW would work for) does not meet the eval contract. The
deployability dimension (0.22) is the out-of-chain fitness axis: it
grades payment-rate affordability against a real regional day-rate
benchmark (not a PDD-declared one) and whether the verification
thresholds are runtime-survivable for real FLWs. It carries a hard-gate
so a faithful-but-undeployable opp cannot clear pass.
This is the per-opp half of the Phase 4 eval pair. See
skills/connect-program-setup-eval/SKILL.md for the program-side
sibling, skills/_eval-template.md for shared contracts, and
skills/eval-calibration/SKILL.md for calibration methodology.
Inputs
Pipeline-only: grades the opportunity a run's Phase 4 configured against that run's PDD; to check a live opportunity, read it with connect-opp-modify's before-read.
| Source | Artifact | Used for |
|---|---|---|
| Phase 1 | 1-design/idea-to-pdd.md | source PDD; Evidence Model + Operational caps drive expectation |
| Phase 4 | 4-connect/connect-opp-setup.md and 4-connect/connect-opp-setup_summary.md | opp config under judgment |
| Phase 4 | 4-connect/connect-program-setup_summary.md | parent Program ID + cross-check on consistency |
| Phase 3 | 3-commcare/app-deploy_summary.md | HQ Deliver app ID + form/module names for cross-check on Deliver Unit wiring |
Products
4-connect/connect-opp-setup-eval_verdict.yaml— verdict YAML per_eval-template.md § Verdict YAML contract. Filename uses this eval skill's name (connect-opp-setup-eval), not the bare producer name — per the 0.12.0 Option-α rule inagents/ace-orchestrator.md § Per-Step Eval Hook.
Process
Four of the six dimensions are graded in CODE; you grade only the two that
need judgment (2026-10-16). verification_flag_fidelity,
payment_unit_fit, deliver_unit_wiring and active_window_status are
diffs of the PDD's ## Program Parameters and the run's own records against
Connect's API state, so lib/connect-opp-config-check.ts computes them.
deployability and archetype_config_coherence (which also owns "did every
PDD Layer A predicate find a home — a form_field_rules row, the CCZ, or a
named [PLATFORM] gap") stay yours. The table below remains the spec for all
six; do not re-grade the four fact dimensions by hand.
-
Gather, save locally: the PDD markdown export,
run_state.yaml, and one JSON file{archetype, opportunity, paymentUnits, createdPaymentUnits, deliverUnits, testUserInvite, payableDeliverSlugs, expectedHqDeliverAppId}—connect_get_opportunity,connect_list_payment_unitsandconnect_list_deliver_unitsat the PM org,connect_list_flw_invitesfor the ACE test phone, theconnect_create_payment_unitsresponse if the opp-setup artifacts recorded it (the list read cannot see amounts, caps or DU wiring — ace#1026), and the payable slugs from3-commcare/app-connect-coverage_deliver.md. A failed read isnull. -
Run the fact half and read its findings:
ACE_ROOT="${CLAUDE_PLUGIN_ROOT:-$(python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json'))); print(d['plugins']['ace@ace'][0]['installPath'])")}" node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/fact-check-eval.ts" connect-opp \ --input <live.json> --pdd <pdd.md> --run-state <run_state.yaml> --target <opp>/<run-id> -
Grade ONLY
deployabilityandarchetype_config_coherenceper the table below; write{deployability: {score, note}, archetype_config_coherence: {score, note}}to a residue JSON (a note is required — say why). -
Re-run with
--residue <residue.json> --out <local verdict.yaml>anddrive_create_filethe result to4-connect/connect-opp-setup-eval_verdict.yaml. The script applies every deduction rule, the deployability hard-gate and the tiers; the verdict recordsmethod: deterministic+judgment-residue.
The numbered steps below are the spec the code (steps 2, 4–7) and your judgment (deployability, coherence) implement.
-
Read inputs from GDrive (paths in
## Inputsabove). Additional sources read on demand:- Connect opp summary:
runs/<run-id>/4-connect/connect-opp-setup_summary.md - Deployment summary:
runs/<run-id>/3-commcare/app-deploy_summary.md(verifies the linked HQ Deliver app + form names match what Connect actually points at).
- Connect opp summary:
-
Detect degraded mode. If the opp-setup artifacts contain
connect_opportunity_id: TBD-MANUAL— i.e., Phase 4 ran in degraded mode because the ace-connect MCPcreate_*tools weren't available — emitverdict: incompleteimmediately with[INFO] degraded-mode artifacts; not gradable as Connect-real. Do not score zero or warn for degraded mode (see sibling rubric for the precedent). -
Extract the PDD's opp-relevant spec. Build a structured expectation:
- Opportunity archetype (
atomic-visit/longitudinal-visits/focus-group/multi-stage). - Verification rules from Evidence Model § Layer A (GPS accuracy, photo presence, consent gate, market-hours window, duplicate detection composite).
- Payment-unit count, type, and structure (per-delivery vs per-session vs per-stage; flat-rate vs piece-rate; derive from Operational caps + Compensation sections).
- Deliver-unit (named slot in the Deliver app, typically the Connectify-tagged form name).
- Active-window duration (Timeline section).
- Opportunity archetype (
-
Extract the built Connect opp config from the opp-setup summary (and via
connect_get_opportunity/connect_list_payment_units/connect_list_deliver_unitsifconnect_opportunity_idis real, not TBD). Setlive_state_verifiedbased on whether those probes succeeded. -
Grade across 6 dimensions. Each dimension is 0–10. Overall score is the weighted mean.
Dimension Weight Criteria Deployability (out-of-chain fitness) 0.22 The out-of-chain anchor. This dimension does NOT grade against the PDD — it grades whether the opp as configured would actually work for real FLWs in the real region. Two sub-axes: (a) Payment-rate affordability. Estimate the per-FLW daily earning (per-unit rate × realistic max-daily-units a real FLW could complete) and compare it to a real-world regional day-rate benchmark for that geography and worker class (gig/CHW field rates for the country/region named in the PDD — use general domain knowledge of local market floors, NOT a PDD-declared figure). If the estimated daily earning is below a plausible local market floor (a rate no real FLW would accept), that is a hard-gate fail (≤3) — the opp will not recruit. If it's implausibly high (>3× a plausible day-rate, signalling a units/currency error), 2-point deduction. (b) Threshold runtime-survivability. Would the verification thresholds wrongly reject legitimate deliveries or pass fraudulent ones in field conditions? A GPS radius too tight for real GPS jitter / tree cover / indoor markets (e.g. ≤10 m hard reject) would bounce legitimate FLWs → 3-point deduction; a radius so loose it can't distinguish the site from a neighbouring one (e.g. >2 km) passes fraud → 3-point deduction. A photo/consent gate with no fallback for a failed upload on a low-bandwidth handset → 2-point deduction. Thresholds that are flatly unworkable for the named field context = hard-gate fail (≤3). Anchors: deployable-as-configured (rate clears local floor, thresholds field-survivable) = 9.0; one soft survivability concern = 6.5; rate below floor OR a threshold class unworkable = ≤3. This dimension is exempt from any deferral carve-out — grade what a deployable opp should contain even where the PDD was silent (PDD silence on a region benchmark is a finding, not a free pass). Verification-flag fidelity 0.20 The PDD's Evidence Model § Layer A specifies hard verification rules (GPS ≤Xm, photo present, consent=yes, market-hours window). Connect's verification flags must enforce the same rules at archetype-appropriate strictness — atomic-visitdefaults to strict GPS + photo;focus-grouprelaxes GPS to room-level + adds attendance check;multi-stageadds stage-gate transitions. Missing a Layer A rule Connect could enforce = 2-point deduction per rule. Adding a rule Connect enforces but PDD doesn't require = 0.5-point deduction. Wrong strictness for the archetype (e.g., GPS strict on a focus-group) = 1.5-point deduction per misfit.Payment-unit fit 0.16 Payment-unit count must match what the PDD declares (one per-delivery PU for atomic-visit; one PU for the WHOLE ARC for longitudinal-visits — the sequence lives in entity_id, not in the PU count, and registration is unpaid; one per-session for focus-group; one PU per stage for multi-stage). Payment type (visit/session/stage/piece) must match. Mismatch in count = 3-point deduction; mismatch in type = 3-point deduction. Do not read alongitudinal-visitsopp against the other three shapes — a single PU there is the CORRECT structure, not a missing per-phase unit (ace#2294: this rubric named only the other three, so a correct configuration read as a mismatch against all of them and drew the deduction). (Rate affordability against a real-world benchmark is graded in Deployability, not here — this dimension is structural-fit only.)Deliver-unit wiring 0.16 The Connect Opportunity must link to the same Deliver Unit name the Deliver app declares (Connectify-tagged form name) on the same HQ project space. Mismatch in deliver-unit name = 4-point deduction (Connect can't credit FLW visits). Wrong HQ app pointer = fail (≤3) — Connect would credit visits against the wrong app. Entity ID composite formula must match what the Deliver app computes (PDD spec ↔ Deliver-app summary ↔ Connect read should agree); divergence between any two = 3-point deduction. Active-window + status 0.16 Active-window duration matches PDD Timeline section ±10%. Status at end of Phase 4 must be activewithis_test=true—connect-opp-setupStep 6.5 activates synchronously by design so Phase 6app-screenshot-capturehas a real signed-in test user, and theis_testflag excludes the opp from prod LLO-facing analytics / payment exports / partner dashboards, so this is NOT premature go-live (real-LLO go-live is Phase 9llo-launch, idempotent on an already-active opp). The fail cases (≤3) are: adraft/never-activated opp at end of Phase 4 (Phase 6 then can't capture screenshots), OR activation withoutis_test=true(a real opp leaking into prod analytics before apps are tested and the bot is gated). Window shorter than the PDD declares (truncates the program's planned duration) is a 2-point deduction; longer than PDD declares (drift, but operationally safe) is a 0.5-point INFO deduction.Archetype-config coherence 0.10 The end-to-end opp config — verification flags + payment unit + deliver-unit + active window — must read coherently for the declared archetype, even when each individual dimension is inside tolerance. Examples: an atomic-visitopp with per-session payment + room-level GPS + facilitator-stipend = incoherent (those are focus-group defaults misapplied). Each cross-dimensional misfit = 2-point deduction. The grader uses anchors: full-archetype-coherent = 9.5; one cross-dim misfit = 7.0; multiple = 4.0; all cross-dims wrong for archetype = 2.0.Deduction rules:
- Any single dimension ≤ 3 → suite verdict
fail, regardless of overall mean. - Deployability hard-gate. If
deployability≤ 3 (rate below the plausible local market floor, or a verification threshold class flatly unworkable for the named field context), the suite verdict isfailregardless of how faithfully the config matches the PDD. A faithful-but-undeployable opp cannot clearpass. Emit a[BLOCKER]. - Inflation guard (mirrors sibling rubrics): if the rubric
surfaces ≥ 2
[WARN]-tierauto_surfacedentries, overall is capped at 8.5.[PLATFORM],[DRIFT],[INFO], and[INFO-SKIPPED]entries are tracked separately and do NOT count toward this guard. - Pre-cap and post-cap reporting per
eval-calibration§ 0.9.4 guidance.
Verdict tiers:
pass— overall ≥ 7.0, no dimension ≤ 3,live_state_verified: true.partial— overall ≥ 7.0, no dimension ≤ 3, artifact looks correct on paper but live MCP probes failed at grading time (network, auth, transient Connect 5xx). Setlive_state_verified: false. Caps overall at 8.5 to mark not-fully-verified.warn— overall ≥ 5.0 < 7.0, or any inflation cap binds.fail— overall < 5.0 OR any dimension ≤ 3.incomplete— degraded-modeTBD-MANUALartifacts, missing PDD or opp-setup summary, or any other structural gap that makes grading impossible.
Severity tiers for
auto_surfacedentries (same set as the sibling rubric):[BLOCKER]— must-fix before merge. Counts as a hard defect.[WARN]— should-fix; counts toward inflation guard.[DRIFT]—connect-opp-setup_summary.mdclaims disagree withconnect_get_opportunity/connect_list_*live read. Diagnostic-only; does NOT deduct (the dimension consuming either source already deducts if either is wrong).[PLATFORM]— defect originates in Connect itself, not the skill's output. Does NOT count toward inflation guard.[INFO]— observational, no action required.
- Any single dimension ≤ 3 → suite verdict
-
Live-state-drift check. When
live_state_verified = true, after grading the dimensions fromconnect-opp-setup_summary.md, compare each summary claim against the liveconnect_get_opportunity/connect_list_payment_units/connect_list_deliver_unitsresponse. Emit a[DRIFT]entry per discrepancy. Drift is diagnostic, not deductive — the dimension score already reflects whichever source is wrong; counting it twice double-penalizes. The drift log is the audit trail downstream investigation reads. -
Write the verdict YAML to
4-connect/connect-opp-setup-eval_verdict.yamlusing the shape fromskills/_eval-template.md § Verdict YAML contract. Dimensions:dimensions: deployability: { weight: 0.22 } # out-of-chain fitness; hard-gate at ≤3 verification_flag_fidelity: { weight: 0.20 } payment_unit_fit: { weight: 0.16 } deliver_unit_wiring: { weight: 0.16 } active_window_status: { weight: 0.16 } archetype_config_coherence: { weight: 0.10 } # weights sum to 1.00Always set
live_state_verifiedbased on whetherconnect_get_*/connect_list_*probes succeeded — false forces verdict ≤partial. -
Auto-surfaced concerns (per
_eval-template.md § Auto-surfaced severity rules, plus skill-specific surfaces):[BLOCKER]whendeployability≤ 3 — payment rate below the plausible local market floor, or a verification threshold class flatly unworkable for the named field context.[WARN]for a soft deployability concern (rate thin but above floor; a threshold survivability concern that degrades but doesn't break field use).[WARN]for each PDD Layer A rule missing from Connect verification flags where Connect could enforce it.[WARN]for deliver-unit name or Entity ID composite mismatch.[WARN]for payment-unit count or type mismatch.[PLATFORM]for Layer A rules Connect cannot enforce today.[DRIFT]for eachconnect-opp-setup_summary.md↔ live-state discrepancy. One entry per field.[INFO]for each over-enforced rule (Connect enforces, PDD doesn't require).[INFO]when the PDD declares no regional day-rate — note thatdeployabilitygraded affordability against an external benchmark instead (PDD silence is a finding, not a skip).
-
Defect-vs-cause discipline. Mirrors
connect-program-setup-eval§ 8: state observations confidently, phrase causes tentatively. UseObserved: <fact>. Likely cause (unverified): <hypothesis>.when both are present. Verified causes (e.g., a follow-upconnect_get_opportunityafter a delay rules out lag) may be stated confidently withverified by <probe>.
LLM-as-Judge Rubric
Calibration target on a non-degraded smoke-run Connect opp-setup:
- Detection rate: ≥ 80% of catalogued Connect-opp-setup issues from
eval-calibration/known-issues.md § Connect opp setup(ground truth TBD — depends on real run, expect ~3–5 catalogued issues). - Inter-run variance: ≤ 0.5 across 3 same-model runs.
- Cross-model variance: ≤ 1.0 for strong calibration.
This rubric ships at provisional until a real non-degraded
opp-setup run produces ground truth. Until then, it correctly emits
incomplete on degraded artifacts.
Archetypes
| Archetype | What this skill does |
|---|---|
atomic-visit | Default. Grades per-delivery payment, GPS-and-photo verification flags at strict defaults, single deliver-unit wiring. |
longitudinal-visits | Grades one payment unit for the whole arc (NOT one per phase) with the sequence carried in entity_id and registration unpaid, plus the longitudinal clause of Layer A. Adds an entity_id-grain sub-check under archetype_config_coherence: the composite must resolve to entity + sequence position, and the grader checks the three failure modes connect-opp-setup § Archetypes enumerates — (1) the case id alone, which folds the whole arc onto one entity row; (2) a concat(username, today())-style key, which silently reverts to cross-sectional grouping; (3) a sequence component that is present but UNCLAMPED, which produces a byte-identical composite and puts every visit past the PDD's per-entity cap on its own key. None of the three stops a payment — Connect pays a repeated key again with the duplicate flag off (ace#2512) — so a PDD per-entity cap is graded as enforced ONLY when a form_field_rules row on the Deliver form's payable_slot field was saved; a record that cites the key alone as the cap's enforcement is a hard deduction. Mode (3) is invisible to a reader of the composite alone, so a record that merely LISTS the components is not evidence — the record must state, per component, whether the PDD's sequence/clamp semantics are implemented (ace#1462, ace#1810). |
focus-group | Grades per-session payment, room-level GPS + attendance-check verification flags, FGD-form deliver-unit wiring. Adds a "facilitator-stipend vs participant-incentive" sub-check under payment_unit_fit. |
multi-stage | Grades per-stage payment structure with stage-gate-aware verification flags. The Stage Gate from the PDD must show up in Connect as a per-stage transition rule. Adds a stage-coverage sub-check under archetype_config_coherence. |
MCP Tools Used
See skills/_eval-template.md § MCP Tools Used (stock) for the Drive
block. Plus:
- ace-connect MCP (when
connect_opportunity_idis real, not TBD):connect_get_opportunity,connect_list_payment_units,connect_list_deliver_units. Skip these calls in degraded mode.
Mode Behavior
See skills/_eval-template.md § Mode Behavior (stock).
Dry-Run Behavior
Per skills/_eval-template.md § Dry-Run Behavior (stock), plus skip
the live connect_get_* / connect_list_* MCP calls (read-only but
treated as offline-unsafe under --dry-run).
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-10-16 | The four fact dimensions are graded in code (lib/connect-opp-config-check.ts, scripts/fact-check-eval.ts connect-opp); the LLM grades only deployability + archetype_config_coherence. Verification fidelity is now what Connect can actually hold — decided form_field_rules vs form_field_rules_saved, JSONPath not XPath (ace#1301), string predicate (ace#2296), live DU id — because gps/duplicate/gps_radius_meters have no Connect form input and are refused (ace#1013); a PDD rule needing them is a coherence question, not a fidelity miss. Payment-unit fit diffs the create response against payment_rate_*, daily_cap_per_flw, total_cap_per_flw, opportunity_currency, with the longitudinal one-PU shape (ace#2294); activation uses checkOppPostcondition (invite-row proof, ace#617). Evidence: an Opus judge re-reading these payloads to grade comparisons (ACE run-efficiency audit, 2026-10-16). | ACE team |
| 2026-05-09 | Initial version. 5 dimensions: verification_flag_fidelity (0.25), payment_unit_fit (0.20), deliver_unit_wiring (0.20), active_window_status (0.20), archetype_config_coherence (0.15). Sibling of connect-program-setup-eval; together they cover Phase 4. Inflation guard at 8.5; same severity tier set ([BLOCKER]/[WARN]/[DRIFT]/[PLATFORM]/[INFO]/[INFO-SKIPPED]). Explicit incomplete verdict for degraded-mode artifacts. Live-state-drift check via connect_get_opportunity + connect_list_payment_units + connect_list_deliver_units. Ships at provisional calibration until a non-degraded run produces ground truth. Closes the "not yet migrated" registry row for connect-opp-setup. | ACE team |
| 2026-05-29 | Add out-of-chain fitness dimension per docs/superpowers/specs/2026-05-29-eval-fitness-gap.md. New deployability (0.22) grades payment-rate affordability against a real regional day-rate benchmark (not a PDD-declared one) and verification-threshold runtime-survivability for real FLWs — with a hard-gate (≤3 → suite fail) so a faithful-but-undeployable opp can't clear pass. Conformance dims reweighted down: verification_flag_fidelity 0.25→0.20, payment_unit_fit 0.20→0.16, deliver_unit_wiring 0.20→0.16, active_window_status 0.20→0.16, archetype_config_coherence 0.15→0.10. Weights sum to 1.00. The old [INFO-SKIPPED] "never deduct" payment-rate sanity sub-check (which conferred no deduction power) is retired — affordability now has teeth inside deployability. | ACE team |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |