Skip to content

dimagi-internal/ace / skills/connect-opp-setup-eval

connect-opp-setup-eval

Unverified

Internal /ace:run step — Grade the run's Connect opportunity setup against its PDD.

Connect Opp Setup Eval

Worker role (multi-role runs): when the dispatch carries a worker_role: {key, label} block, scope this run to that role's apps and its part of the PDD, and read and write every 3-commcare/ (or 4-connect/) file at the same name under 3-commcare/roles/<key>/ (4-connect/roles/<key>/). No block = a single-role run, unchanged. Contract: skills/_worker-roles.md.

connect-opp-setup is the Phase 4 skill that creates the Connect Opportunity that hangs off the Program — verification flags, payment units, deliver-unit links, active window. Where connect-program-setup grades the program-side translation, this rubric grades the opportunity-side. Together they cover the full Phase 4 Connect-side build.

Five of the six dimensions triangulate PDD ↔ Connect-config ↔ Deliver-summary — that's fidelity to the AI authoring chain, not fitness of the opportunity to deploy. Per skills/_eval-template.md § The out-of-chain fitness requirement, a rubric whose every dimension is satisfiable by a config that matches the PDD but would wrongly reject legitimate FLW deliveries (or pay a rate no real FLW would work for) does not meet the eval contract. The deployability dimension (0.22) is the out-of-chain fitness axis: it grades payment-rate affordability against a real regional day-rate benchmark (not a PDD-declared one) and whether the verification thresholds are runtime-survivable for real FLWs. It carries a hard-gate so a faithful-but-undeployable opp cannot clear pass.

This is the per-opp half of the Phase 4 eval pair. See skills/connect-program-setup-eval/SKILL.md for the program-side sibling, skills/_eval-template.md for shared contracts, and skills/eval-calibration/SKILL.md for calibration methodology.

Inputs

Pipeline-only: grades the opportunity a run's Phase 4 configured against that run's PDD; to check a live opportunity, read it with connect-opp-modify's before-read.

SourceArtifactUsed for
Phase 11-design/idea-to-pdd.mdsource PDD; Evidence Model + Operational caps drive expectation
Phase 44-connect/connect-opp-setup.md and 4-connect/connect-opp-setup_summary.mdopp config under judgment
Phase 44-connect/connect-program-setup_summary.mdparent Program ID + cross-check on consistency
Phase 33-commcare/app-deploy_summary.mdHQ Deliver app ID + form/module names for cross-check on Deliver Unit wiring

Products

  • 4-connect/connect-opp-setup-eval_verdict.yaml — verdict YAML per _eval-template.md § Verdict YAML contract. Filename uses this eval skill's name (connect-opp-setup-eval), not the bare producer name — per the 0.12.0 Option-α rule in agents/ace-orchestrator.md § Per-Step Eval Hook.

Process

Four of the six dimensions are graded in CODE; you grade only the two that need judgment (2026-10-16). verification_flag_fidelity, payment_unit_fit, deliver_unit_wiring and active_window_status are diffs of the PDD's ## Program Parameters and the run's own records against Connect's API state, so lib/connect-opp-config-check.ts computes them. deployability and archetype_config_coherence (which also owns "did every PDD Layer A predicate find a home — a form_field_rules row, the CCZ, or a named [PLATFORM] gap") stay yours. The table below remains the spec for all six; do not re-grade the four fact dimensions by hand.

  1. Gather, save locally: the PDD markdown export, run_state.yaml, and one JSON file {archetype, opportunity, paymentUnits, createdPaymentUnits, deliverUnits, testUserInvite, payableDeliverSlugs, expectedHqDeliverAppId} — connect_get_opportunity, connect_list_payment_units and connect_list_deliver_units at the PM org, connect_list_flw_invites for the ACE test phone, the connect_create_payment_units response if the opp-setup artifacts recorded it (the list read cannot see amounts, caps or DU wiring — ace#1026), and the payable slugs from 3-commcare/app-connect-coverage_deliver.md. A failed read is null.

  2. Run the fact half and read its findings:

    ACE_ROOT="${CLAUDE_PLUGIN_ROOT:-$(python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json'))); print(d['plugins']['ace@ace'][0]['installPath'])")}"
    node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/fact-check-eval.ts" connect-opp \
      --input <live.json> --pdd <pdd.md> --run-state <run_state.yaml> --target <opp>/<run-id>
    
  3. Grade ONLY deployability and archetype_config_coherence per the table below; write {deployability: {score, note}, archetype_config_coherence: {score, note}} to a residue JSON (a note is required — say why).

  4. Re-run with --residue <residue.json> --out <local verdict.yaml> and drive_create_file the result to 4-connect/connect-opp-setup-eval_verdict.yaml. The script applies every deduction rule, the deployability hard-gate and the tiers; the verdict records method: deterministic+judgment-residue.

The numbered steps below are the spec the code (steps 2, 4–7) and your judgment (deployability, coherence) implement.

  1. Read inputs from GDrive (paths in ## Inputs above). Additional sources read on demand:

    • Connect opp summary: runs/<run-id>/4-connect/connect-opp-setup_summary.md
    • Deployment summary: runs/<run-id>/3-commcare/app-deploy_summary.md (verifies the linked HQ Deliver app + form names match what Connect actually points at).
  2. Detect degraded mode. If the opp-setup artifacts contain connect_opportunity_id: TBD-MANUAL — i.e., Phase 4 ran in degraded mode because the ace-connect MCP create_* tools weren't available — emit verdict: incomplete immediately with [INFO] degraded-mode artifacts; not gradable as Connect-real. Do not score zero or warn for degraded mode (see sibling rubric for the precedent).

  3. Extract the PDD's opp-relevant spec. Build a structured expectation:

    • Opportunity archetype (atomic-visit / longitudinal-visits / focus-group / multi-stage).
    • Verification rules from Evidence Model § Layer A (GPS accuracy, photo presence, consent gate, market-hours window, duplicate detection composite).
    • Payment-unit count, type, and structure (per-delivery vs per-session vs per-stage; flat-rate vs piece-rate; derive from Operational caps + Compensation sections).
    • Deliver-unit (named slot in the Deliver app, typically the Connectify-tagged form name).
    • Active-window duration (Timeline section).
  4. Extract the built Connect opp config from the opp-setup summary (and via connect_get_opportunity / connect_list_payment_units / connect_list_deliver_units if connect_opportunity_id is real, not TBD). Set live_state_verified based on whether those probes succeeded.

  5. Grade across 6 dimensions. Each dimension is 0–10. Overall score is the weighted mean.

    DimensionWeightCriteria
    Deployability (out-of-chain fitness)0.22The out-of-chain anchor. This dimension does NOT grade against the PDD — it grades whether the opp as configured would actually work for real FLWs in the real region. Two sub-axes: (a) Payment-rate affordability. Estimate the per-FLW daily earning (per-unit rate × realistic max-daily-units a real FLW could complete) and compare it to a real-world regional day-rate benchmark for that geography and worker class (gig/CHW field rates for the country/region named in the PDD — use general domain knowledge of local market floors, NOT a PDD-declared figure). If the estimated daily earning is below a plausible local market floor (a rate no real FLW would accept), that is a hard-gate fail (≤3) — the opp will not recruit. If it's implausibly high (>3× a plausible day-rate, signalling a units/currency error), 2-point deduction. (b) Threshold runtime-survivability. Would the verification thresholds wrongly reject legitimate deliveries or pass fraudulent ones in field conditions? A GPS radius too tight for real GPS jitter / tree cover / indoor markets (e.g. ≤10 m hard reject) would bounce legitimate FLWs → 3-point deduction; a radius so loose it can't distinguish the site from a neighbouring one (e.g. >2 km) passes fraud → 3-point deduction. A photo/consent gate with no fallback for a failed upload on a low-bandwidth handset → 2-point deduction. Thresholds that are flatly unworkable for the named field context = hard-gate fail (≤3). Anchors: deployable-as-configured (rate clears local floor, thresholds field-survivable) = 9.0; one soft survivability concern = 6.5; rate below floor OR a threshold class unworkable = ≤3. This dimension is exempt from any deferral carve-out — grade what a deployable opp should contain even where the PDD was silent (PDD silence on a region benchmark is a finding, not a free pass).
    Verification-flag fidelity0.20The PDD's Evidence Model § Layer A specifies hard verification rules (GPS ≤Xm, photo present, consent=yes, market-hours window). Connect's verification flags must enforce the same rules at archetype-appropriate strictness — atomic-visit defaults to strict GPS + photo; focus-group relaxes GPS to room-level + adds attendance check; multi-stage adds stage-gate transitions. Missing a Layer A rule Connect could enforce = 2-point deduction per rule. Adding a rule Connect enforces but PDD doesn't require = 0.5-point deduction. Wrong strictness for the archetype (e.g., GPS strict on a focus-group) = 1.5-point deduction per misfit.
    Payment-unit fit0.16Payment-unit count must match what the PDD declares (one per-delivery PU for atomic-visit; one PU for the WHOLE ARC for longitudinal-visits — the sequence lives in entity_id, not in the PU count, and registration is unpaid; one per-session for focus-group; one PU per stage for multi-stage). Payment type (visit/session/stage/piece) must match. Mismatch in count = 3-point deduction; mismatch in type = 3-point deduction. Do not read a longitudinal-visits opp against the other three shapes — a single PU there is the CORRECT structure, not a missing per-phase unit (ace#2294: this rubric named only the other three, so a correct configuration read as a mismatch against all of them and drew the deduction). (Rate affordability against a real-world benchmark is graded in Deployability, not here — this dimension is structural-fit only.)
    Deliver-unit wiring0.16The Connect Opportunity must link to the same Deliver Unit name the Deliver app declares (Connectify-tagged form name) on the same HQ project space. Mismatch in deliver-unit name = 4-point deduction (Connect can't credit FLW visits). Wrong HQ app pointer = fail (≤3) — Connect would credit visits against the wrong app. Entity ID composite formula must match what the Deliver app computes (PDD spec ↔ Deliver-app summary ↔ Connect read should agree); divergence between any two = 3-point deduction.
    Active-window + status0.16Active-window duration matches PDD Timeline section ±10%. Status at end of Phase 4 must be active with is_test=true — connect-opp-setup Step 6.5 activates synchronously by design so Phase 6 app-screenshot-capture has a real signed-in test user, and the is_test flag excludes the opp from prod LLO-facing analytics / payment exports / partner dashboards, so this is NOT premature go-live (real-LLO go-live is Phase 9 llo-launch, idempotent on an already-active opp). The fail cases (≤3) are: a draft/never-activated opp at end of Phase 4 (Phase 6 then can't capture screenshots), OR activation without is_test=true (a real opp leaking into prod analytics before apps are tested and the bot is gated). Window shorter than the PDD declares (truncates the program's planned duration) is a 2-point deduction; longer than PDD declares (drift, but operationally safe) is a 0.5-point INFO deduction.
    Archetype-config coherence0.10The end-to-end opp config — verification flags + payment unit + deliver-unit + active window — must read coherently for the declared archetype, even when each individual dimension is inside tolerance. Examples: an atomic-visit opp with per-session payment + room-level GPS + facilitator-stipend = incoherent (those are focus-group defaults misapplied). Each cross-dimensional misfit = 2-point deduction. The grader uses anchors: full-archetype-coherent = 9.5; one cross-dim misfit = 7.0; multiple = 4.0; all cross-dims wrong for archetype = 2.0.

    Deduction rules:

    • Any single dimension ≤ 3 → suite verdict fail, regardless of overall mean.
    • Deployability hard-gate. If deployability ≤ 3 (rate below the plausible local market floor, or a verification threshold class flatly unworkable for the named field context), the suite verdict is fail regardless of how faithfully the config matches the PDD. A faithful-but-undeployable opp cannot clear pass. Emit a [BLOCKER].
    • Inflation guard (mirrors sibling rubrics): if the rubric surfaces ≥ 2 [WARN]-tier auto_surfaced entries, overall is capped at 8.5. [PLATFORM], [DRIFT], [INFO], and [INFO-SKIPPED] entries are tracked separately and do NOT count toward this guard.
    • Pre-cap and post-cap reporting per eval-calibration § 0.9.4 guidance.

    Verdict tiers:

    • pass — overall ≥ 7.0, no dimension ≤ 3, live_state_verified: true.
    • partial — overall ≥ 7.0, no dimension ≤ 3, artifact looks correct on paper but live MCP probes failed at grading time (network, auth, transient Connect 5xx). Set live_state_verified: false. Caps overall at 8.5 to mark not-fully-verified.
    • warn — overall ≥ 5.0 < 7.0, or any inflation cap binds.
    • fail — overall < 5.0 OR any dimension ≤ 3.
    • incomplete — degraded-mode TBD-MANUAL artifacts, missing PDD or opp-setup summary, or any other structural gap that makes grading impossible.

    Severity tiers for auto_surfaced entries (same set as the sibling rubric):

    • [BLOCKER] — must-fix before merge. Counts as a hard defect.
    • [WARN] — should-fix; counts toward inflation guard.
    • [DRIFT] — connect-opp-setup_summary.md claims disagree with connect_get_opportunity / connect_list_* live read. Diagnostic-only; does NOT deduct (the dimension consuming either source already deducts if either is wrong).
    • [PLATFORM] — defect originates in Connect itself, not the skill's output. Does NOT count toward inflation guard.
    • [INFO] — observational, no action required.
  6. Live-state-drift check. When live_state_verified = true, after grading the dimensions from connect-opp-setup_summary.md, compare each summary claim against the live connect_get_opportunity / connect_list_payment_units / connect_list_deliver_units response. Emit a [DRIFT] entry per discrepancy. Drift is diagnostic, not deductive — the dimension score already reflects whichever source is wrong; counting it twice double-penalizes. The drift log is the audit trail downstream investigation reads.

  7. Write the verdict YAML to 4-connect/connect-opp-setup-eval_verdict.yaml using the shape from skills/_eval-template.md § Verdict YAML contract. Dimensions:

    dimensions:
      deployability:                 { weight: 0.22 }   # out-of-chain fitness; hard-gate at ≤3
      verification_flag_fidelity:    { weight: 0.20 }
      payment_unit_fit:              { weight: 0.16 }
      deliver_unit_wiring:           { weight: 0.16 }
      active_window_status:          { weight: 0.16 }
      archetype_config_coherence:    { weight: 0.10 }
    # weights sum to 1.00
    

    Always set live_state_verified based on whether connect_get_* / connect_list_* probes succeeded — false forces verdict ≤ partial.

  8. Auto-surfaced concerns (per _eval-template.md § Auto-surfaced severity rules, plus skill-specific surfaces):

    • [BLOCKER] when deployability ≤ 3 — payment rate below the plausible local market floor, or a verification threshold class flatly unworkable for the named field context.
    • [WARN] for a soft deployability concern (rate thin but above floor; a threshold survivability concern that degrades but doesn't break field use).
    • [WARN] for each PDD Layer A rule missing from Connect verification flags where Connect could enforce it.
    • [WARN] for deliver-unit name or Entity ID composite mismatch.
    • [WARN] for payment-unit count or type mismatch.
    • [PLATFORM] for Layer A rules Connect cannot enforce today.
    • [DRIFT] for each connect-opp-setup_summary.md ↔ live-state discrepancy. One entry per field.
    • [INFO] for each over-enforced rule (Connect enforces, PDD doesn't require).
    • [INFO] when the PDD declares no regional day-rate — note that deployability graded affordability against an external benchmark instead (PDD silence is a finding, not a skip).
  9. Defect-vs-cause discipline. Mirrors connect-program-setup-eval § 8: state observations confidently, phrase causes tentatively. Use Observed: <fact>. Likely cause (unverified): <hypothesis>. when both are present. Verified causes (e.g., a follow-up connect_get_opportunity after a delay rules out lag) may be stated confidently with verified by <probe>.

LLM-as-Judge Rubric

Calibration target on a non-degraded smoke-run Connect opp-setup:

  • Detection rate: ≥ 80% of catalogued Connect-opp-setup issues from eval-calibration/known-issues.md § Connect opp setup (ground truth TBD — depends on real run, expect ~3–5 catalogued issues).
  • Inter-run variance: ≤ 0.5 across 3 same-model runs.
  • Cross-model variance: ≤ 1.0 for strong calibration.

This rubric ships at provisional until a real non-degraded opp-setup run produces ground truth. Until then, it correctly emits incomplete on degraded artifacts.

Archetypes

ArchetypeWhat this skill does
atomic-visitDefault. Grades per-delivery payment, GPS-and-photo verification flags at strict defaults, single deliver-unit wiring.
longitudinal-visitsGrades one payment unit for the whole arc (NOT one per phase) with the sequence carried in entity_id and registration unpaid, plus the longitudinal clause of Layer A. Adds an entity_id-grain sub-check under archetype_config_coherence: the composite must resolve to entity + sequence position, and the grader checks the three failure modes connect-opp-setup § Archetypes enumerates — (1) the case id alone, which folds the whole arc onto one entity row; (2) a concat(username, today())-style key, which silently reverts to cross-sectional grouping; (3) a sequence component that is present but UNCLAMPED, which produces a byte-identical composite and puts every visit past the PDD's per-entity cap on its own key. None of the three stops a payment — Connect pays a repeated key again with the duplicate flag off (ace#2512) — so a PDD per-entity cap is graded as enforced ONLY when a form_field_rules row on the Deliver form's payable_slot field was saved; a record that cites the key alone as the cap's enforcement is a hard deduction. Mode (3) is invisible to a reader of the composite alone, so a record that merely LISTS the components is not evidence — the record must state, per component, whether the PDD's sequence/clamp semantics are implemented (ace#1462, ace#1810).
focus-groupGrades per-session payment, room-level GPS + attendance-check verification flags, FGD-form deliver-unit wiring. Adds a "facilitator-stipend vs participant-incentive" sub-check under payment_unit_fit.
multi-stageGrades per-stage payment structure with stage-gate-aware verification flags. The Stage Gate from the PDD must show up in Connect as a per-stage transition rule. Adds a stage-coverage sub-check under archetype_config_coherence.

MCP Tools Used

See skills/_eval-template.md § MCP Tools Used (stock) for the Drive block. Plus:

  • ace-connect MCP (when connect_opportunity_id is real, not TBD): connect_get_opportunity, connect_list_payment_units, connect_list_deliver_units. Skip these calls in degraded mode.

Mode Behavior

See skills/_eval-template.md § Mode Behavior (stock).

Dry-Run Behavior

Per skills/_eval-template.md § Dry-Run Behavior (stock), plus skip the live connect_get_* / connect_list_* MCP calls (read-only but treated as offline-unsafe under --dry-run).

Change Log

DateChangeAuthor
2026-10-16The four fact dimensions are graded in code (lib/connect-opp-config-check.ts, scripts/fact-check-eval.ts connect-opp); the LLM grades only deployability + archetype_config_coherence. Verification fidelity is now what Connect can actually hold — decided form_field_rules vs form_field_rules_saved, JSONPath not XPath (ace#1301), string predicate (ace#2296), live DU id — because gps/duplicate/gps_radius_meters have no Connect form input and are refused (ace#1013); a PDD rule needing them is a coherence question, not a fidelity miss. Payment-unit fit diffs the create response against payment_rate_*, daily_cap_per_flw, total_cap_per_flw, opportunity_currency, with the longitudinal one-PU shape (ace#2294); activation uses checkOppPostcondition (invite-row proof, ace#617). Evidence: an Opus judge re-reading these payloads to grade comparisons (ACE run-efficiency audit, 2026-10-16).ACE team
2026-05-09Initial version. 5 dimensions: verification_flag_fidelity (0.25), payment_unit_fit (0.20), deliver_unit_wiring (0.20), active_window_status (0.20), archetype_config_coherence (0.15). Sibling of connect-program-setup-eval; together they cover Phase 4. Inflation guard at 8.5; same severity tier set ([BLOCKER]/[WARN]/[DRIFT]/[PLATFORM]/[INFO]/[INFO-SKIPPED]). Explicit incomplete verdict for degraded-mode artifacts. Live-state-drift check via connect_get_opportunity + connect_list_payment_units + connect_list_deliver_units. Ships at provisional calibration until a non-degraded run produces ground truth. Closes the "not yet migrated" registry row for connect-opp-setup.ACE team
2026-05-29Add out-of-chain fitness dimension per docs/superpowers/specs/2026-05-29-eval-fitness-gap.md. New deployability (0.22) grades payment-rate affordability against a real regional day-rate benchmark (not a PDD-declared one) and verification-threshold runtime-survivability for real FLWs — with a hard-gate (≤3 → suite fail) so a faithful-but-undeployable opp can't clear pass. Conformance dims reweighted down: verification_flag_fidelity 0.25→0.20, payment_unit_fit 0.20→0.16, deliver_unit_wiring 0.20→0.16, active_window_status 0.20→0.16, archetype_config_coherence 0.15→0.10. Weights sum to 1.00. The old [INFO-SKIPPED] "never deduct" payment-rate sanity sub-check (which conferred no deduction power) is retired — affordability now has teeth inside deployability.ACE team
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 192kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 192kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 192kUpdated Claude CodeCodexVerifiedInstall