dimagi-internal/ace / skills/connect-opp-setup-eval
connect-opp-setup-eval
VerifiedInternal /ace:run step — Grade the run's Connect opportunity setup against its PDD.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 23.2 KB |
SKILL.md
View on GitHub---
name: connect-opp-setup-eval
description: >
Internal /ace:run step — Grade the run's Connect opportunity setup against its PDD.
user-invocable: false
disable-model-invocation: false
---
# Connect Opp Setup Eval
**Worker role (multi-role runs):** when the dispatch carries a `worker_role: {key, label}` block, scope this run to that role's apps and its part of the PDD, and read and write every `3-commcare/` (or `4-connect/`) file at the same name under `3-commcare/roles/<key>/` (`4-connect/roles/<key>/`). No block = a single-role run, unchanged. Contract: [`skills/_worker-roles.md`](../_worker-roles.md).
`connect-opp-setup` is the Phase 4 skill that creates the Connect
Opportunity that hangs off the Program — verification flags, payment
units, deliver-unit links, active window. Where `connect-program-setup`
grades the program-side translation, this rubric grades the
opportunity-side. Together they cover the full Phase 4 Connect-side
build.
Five of the six dimensions triangulate PDD ↔ Connect-config ↔
Deliver-summary — that's **fidelity to the AI authoring chain**, not
**fitness of the opportunity to deploy**. Per
`skills/_eval-template.md § The out-of-chain fitness requirement`, a
rubric whose every dimension is satisfiable by a config that matches the
PDD but would wrongly reject legitimate FLW deliveries (or pay a rate no
real FLW would work for) does not meet the eval contract. The
**`deployability`** dimension (0.22) is the out-of-chain fitness axis: it
grades payment-rate affordability against a *real regional day-rate
benchmark* (not a PDD-declared one) and whether the verification
thresholds are runtime-survivable for real FLWs. It carries a hard-gate
so a faithful-but-undeployable opp cannot clear `pass`.
This is the per-opp half of the Phase 4 eval pair. See
`skills/connect-program-setup-eval/SKILL.md` for the program-side
sibling, `skills/_eval-template.md` for shared contracts, and
`skills/eval-calibration/SKILL.md` for calibration methodology.
## Inputs
Pipeline-only: grades the opportunity a run's Phase 4 configured against that run's PDD; to check a live opportunity, read it with `connect-opp-modify`'s before-read.
| Source | Artifact | Used for |
|---|---|---|
| Phase 1 | `1-design/idea-to-pdd.md` | source PDD; Evidence Model + Operational caps drive expectation |
| Phase 4 | `4-connect/connect-opp-setup.md` and `4-connect/connect-opp-setup_summary.md` | opp config under judgment |
| Phase 4 | `4-connect/connect-program-setup_summary.md` | parent Program ID + cross-check on consistency |
| Phase 3 | `3-commcare/app-deploy_summary.md` | HQ Deliver app ID + form/module names for cross-check on Deliver Unit wiring |
## Products
- `4-connect/connect-opp-setup-eval_verdict.yaml` — verdict YAML per `_eval-template.md § Verdict YAML contract`. Filename uses **this eval skill's** name (`connect-opp-setup-eval`), not the bare producer name — per the 0.12.0 Option-α rule in `agents/ace-orchestrator.md § Per-Step Eval Hook`.
## Process
**Four of the six dimensions are graded in CODE; you grade only the two that
need judgment (2026-10-16).** `verification_flag_fidelity`,
`payment_unit_fit`, `deliver_unit_wiring` and `active_window_status` are
diffs of the PDD's `## Program Parameters` and the run's own records against
Connect's API state, so `lib/connect-opp-config-check.ts` computes them.
`deployability` and `archetype_config_coherence` (which also owns "did every
PDD Layer A predicate find a home — a `form_field_rules` row, the CCZ, or a
named `[PLATFORM]` gap") stay yours. The table below remains the spec for all
six; do not re-grade the four fact dimensions by hand.
1. Gather, save locally: the PDD markdown export, `run_state.yaml`, and one
JSON file `{archetype, opportunity, paymentUnits, createdPaymentUnits,
deliverUnits, testUserInvite, payableDeliverSlugs, expectedHqDeliverAppId}` —
`connect_get_opportunity`, `connect_list_payment_units` and
`connect_list_deliver_units` at the PM org, `connect_list_flw_invites` for
the ACE test phone, the `connect_create_payment_units` response if the
opp-setup artifacts recorded it (the list read cannot see amounts, caps or
DU wiring — ace#1026), and the payable slugs from
`3-commcare/app-connect-coverage_deliver.md`. A failed read is `null`.
2. Run the fact half and read its findings:
```bash
ACE_ROOT="${CLAUDE_PLUGIN_ROOT:-$(python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json'))); print(d['plugins']['ace@ace'][0]['installPath'])")}"
node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/fact-check-eval.ts" connect-opp \
--input <live.json> --pdd <pdd.md> --run-state <run_state.yaml> --target <opp>/<run-id>
```
3. Grade ONLY `deployability` and `archetype_config_coherence` per the table
below; write `{deployability: {score, note}, archetype_config_coherence:
{score, note}}` to a residue JSON (a note is required — say why).
4. Re-run with `--residue <residue.json> --out <local verdict.yaml>` and
`drive_create_file` the result to
`4-connect/connect-opp-setup-eval_verdict.yaml`. The script applies every
deduction rule, the deployability hard-gate and the tiers; the verdict
records `method: deterministic+judgment-residue`.
The numbered steps below are the spec the code (steps 2, 4–7) and your
judgment (deployability, coherence) implement.
1. **Read inputs from GDrive** (paths in `## Inputs` above).
Additional sources read on demand:
- Connect opp summary: `runs/<run-id>/4-connect/connect-opp-setup_summary.md`
- Deployment summary: `runs/<run-id>/3-commcare/app-deploy_summary.md`
(verifies the linked HQ Deliver app + form names match what
Connect actually points at).
2. **Detect degraded mode.** If the opp-setup artifacts contain
`connect_opportunity_id: TBD-MANUAL` — i.e., Phase 4 ran in degraded
mode because the ace-connect MCP `create_*` tools weren't available
— emit `verdict: incomplete` immediately with `[INFO] degraded-mode
artifacts; not gradable as Connect-real`. Do not score zero or warn
for degraded mode (see sibling rubric for the precedent).
3. **Extract the PDD's opp-relevant spec.** Build a structured
expectation:
- Opportunity archetype (`atomic-visit` / `longitudinal-visits` /
`focus-group` / `multi-stage`).
- Verification rules from Evidence Model § Layer A (GPS accuracy,
photo presence, consent gate, market-hours window, duplicate
detection composite).
- Payment-unit count, type, and structure (per-delivery vs
per-session vs per-stage; flat-rate vs piece-rate; derive from
Operational caps + Compensation sections).
- Deliver-unit (named slot in the Deliver app, typically the
Connectify-tagged form name).
- Active-window duration (Timeline section).
4. **Extract the built Connect opp config** from the opp-setup summary
(and via `connect_get_opportunity` / `connect_list_payment_units` /
`connect_list_deliver_units` if `connect_opportunity_id` is real,
not TBD). Set `live_state_verified` based on whether those probes
succeeded.
5. **Grade across 6 dimensions.** Each dimension is 0–10. Overall
score is the weighted mean.
| Dimension | Weight | Criteria |
|---|---|---|
| **Deployability (out-of-chain fitness)** | 0.22 | The out-of-chain anchor. This dimension does NOT grade against the PDD — it grades whether the opp as configured would actually work for real FLWs in the real region. Two sub-axes: **(a) Payment-rate affordability.** Estimate the per-FLW daily earning (per-unit rate × realistic max-daily-units a real FLW could complete) and compare it to a *real-world regional day-rate benchmark* for that geography and worker class (gig/CHW field rates for the country/region named in the PDD — use general domain knowledge of local market floors, NOT a PDD-declared figure). If the estimated daily earning is **below a plausible local market floor** (a rate no real FLW would accept), that is a **hard-gate fail (≤3)** — the opp will not recruit. If it's implausibly high (>3× a plausible day-rate, signalling a units/currency error), 2-point deduction. **(b) Threshold runtime-survivability.** Would the verification thresholds wrongly reject legitimate deliveries or pass fraudulent ones in field conditions? A GPS radius too tight for real GPS jitter / tree cover / indoor markets (e.g. ≤10 m hard reject) would bounce legitimate FLWs → 3-point deduction; a radius so loose it can't distinguish the site from a neighbouring one (e.g. >2 km) passes fraud → 3-point deduction. A photo/consent gate with no fallback for a failed upload on a low-bandwidth handset → 2-point deduction. Thresholds that are flatly unworkable for the named field context = **hard-gate fail (≤3)**. Anchors: deployable-as-configured (rate clears local floor, thresholds field-survivable) = **9.0**; one soft survivability concern = **6.5**; rate below floor OR a threshold class unworkable = **≤3**. This dimension is exempt from any deferral carve-out — grade what a *deployable* opp should contain even where the PDD was silent (PDD silence on a region benchmark is a finding, not a free pass). |
| **Verification-flag fidelity** | 0.20 | The PDD's Evidence Model § Layer A specifies hard verification rules (GPS ≤Xm, photo present, consent=yes, market-hours window). Connect's verification flags must enforce the same rules at archetype-appropriate strictness — `atomic-visit` defaults to strict GPS + photo; `focus-group` relaxes GPS to room-level + adds attendance check; `multi-stage` adds stage-gate transitions. Missing a Layer A rule Connect could enforce = 2-point deduction per rule. Adding a rule Connect enforces but PDD doesn't require = 0.5-point deduction. Wrong strictness for the archetype (e.g., GPS strict on a focus-group) = 1.5-point deduction per misfit. |
| **Payment-unit fit** | 0.16 | Payment-unit *count* must match what the PDD declares (one per-delivery PU for atomic-visit; **one PU for the WHOLE ARC for longitudinal-visits** — the sequence lives in `entity_id`, not in the PU count, and registration is unpaid; one per-session for focus-group; one PU per stage for multi-stage). Payment *type* (visit/session/stage/piece) must match. Mismatch in count = 3-point deduction; mismatch in type = 3-point deduction. **Do not read a `longitudinal-visits` opp against the other three shapes** — a single PU there is the CORRECT structure, not a missing per-phase unit (ace#2294: this rubric named only the other three, so a correct configuration read as a mismatch against all of them and drew the deduction). (Rate *affordability* against a real-world benchmark is graded in **Deployability**, not here — this dimension is structural-fit only.) |
| **Deliver-unit wiring** | 0.16 | The Connect Opportunity must link to the same Deliver Unit name the Deliver app declares (Connectify-tagged form name) on the same HQ project space. Mismatch in deliver-unit name = 4-point deduction (Connect can't credit FLW visits). Wrong HQ app pointer = fail (≤3) — Connect would credit visits against the wrong app. Entity ID composite formula must match what the Deliver app computes (PDD spec ↔ Deliver-app summary ↔ Connect read should agree); divergence between any two = 3-point deduction. |
| **Active-window + status** | 0.16 | Active-window duration matches PDD Timeline section ±10%. Status at end of Phase 4 must be `active` **with `is_test=true`** — `connect-opp-setup` Step 6.5 activates synchronously by design so Phase 6 `app-screenshot-capture` has a real signed-in test user, and the `is_test` flag excludes the opp from prod LLO-facing analytics / payment exports / partner dashboards, so this is NOT premature go-live (real-LLO go-live is Phase 9 `llo-launch`, idempotent on an already-active opp). The fail cases (≤3) are: a `draft`/never-activated opp at end of Phase 4 (Phase 6 then can't capture screenshots), OR activation **without** `is_test=true` (a real opp leaking into prod analytics before apps are tested and the bot is gated). Window shorter than the PDD declares (truncates the program's planned duration) is a 2-point deduction; longer than PDD declares (drift, but operationally safe) is a 0.5-point INFO deduction. |
| **Archetype-config coherence** | 0.10 | The end-to-end opp config — verification flags + payment unit + deliver-unit + active window — must read coherently for the declared archetype, even when each individual dimension is inside tolerance. Examples: an `atomic-visit` opp with per-session payment + room-level GPS + facilitator-stipend = incoherent (those are focus-group defaults misapplied). Each cross-dimensional misfit = 2-point deduction. The grader uses anchors: full-archetype-coherent = **9.5**; one cross-dim misfit = **7.0**; multiple = **4.0**; all cross-dims wrong for archetype = **2.0**. |
**Deduction rules:**
- Any single dimension ≤ 3 → suite verdict `fail`, regardless of
overall mean.
- **Deployability hard-gate.** If `deployability` ≤ 3 (rate below the
plausible local market floor, or a verification threshold class
flatly unworkable for the named field context), the suite verdict
is `fail` regardless of how faithfully the config matches the PDD.
A faithful-but-undeployable opp cannot clear `pass`. Emit a
`[BLOCKER]`.
- **Inflation guard (mirrors sibling rubrics):** if the rubric
surfaces ≥ 2 `[WARN]`-tier `auto_surfaced` entries, overall is
capped at **8.5**. `[PLATFORM]`, `[DRIFT]`, `[INFO]`, and
`[INFO-SKIPPED]` entries are tracked separately and do NOT count
toward this guard.
- **Pre-cap and post-cap reporting** per `eval-calibration` § 0.9.4
guidance.
**Verdict tiers:**
- `pass` — overall ≥ 7.0, no dimension ≤ 3, `live_state_verified: true`.
- `partial` — overall ≥ 7.0, no dimension ≤ 3, **artifact looks
correct on paper but live MCP probes failed at grading time**
(network, auth, transient Connect 5xx). Set `live_state_verified:
false`. Caps overall at **8.5** to mark not-fully-verified.
- `warn` — overall ≥ 5.0 < 7.0, or any inflation cap binds.
- `fail` — overall < 5.0 OR any dimension ≤ 3.
- `incomplete` — degraded-mode `TBD-MANUAL` artifacts, missing PDD
or opp-setup summary, or any other structural gap that makes
grading impossible.
**Severity tiers** for `auto_surfaced` entries (same set as the
sibling rubric):
- `[BLOCKER]` — must-fix before merge. Counts as a hard defect.
- `[WARN]` — should-fix; counts toward inflation guard.
- `[DRIFT]` — `connect-opp-setup_summary.md` claims disagree with
`connect_get_opportunity` / `connect_list_*` live read.
Diagnostic-only; does NOT deduct (the dimension consuming either
source already deducts if either is wrong).
- `[PLATFORM]` — defect originates in Connect itself, not the
skill's output. Does NOT count toward inflation guard.
- `[INFO]` — observational, no action required.
6. **Live-state-drift check.** When `live_state_verified = true`, after
grading the dimensions from `connect-opp-setup_summary.md`, compare
each summary claim against the live `connect_get_opportunity` /
`connect_list_payment_units` / `connect_list_deliver_units` response.
Emit a `[DRIFT]` entry per discrepancy. Drift is diagnostic, not
deductive — the dimension score already reflects whichever source is
wrong; counting it twice double-penalizes. The drift log is the
audit trail downstream investigation reads.
7. **Write the verdict YAML** to
`4-connect/connect-opp-setup-eval_verdict.yaml` using the shape from
`skills/_eval-template.md § Verdict YAML contract`. Dimensions:
```yaml
dimensions:
deployability: { weight: 0.22 } # out-of-chain fitness; hard-gate at ≤3
verification_flag_fidelity: { weight: 0.20 }
payment_unit_fit: { weight: 0.16 }
deliver_unit_wiring: { weight: 0.16 }
active_window_status: { weight: 0.16 }
archetype_config_coherence: { weight: 0.10 }
# weights sum to 1.00
```
Always set `live_state_verified` based on whether `connect_get_*` /
`connect_list_*` probes succeeded — false forces verdict ≤ `partial`.
8. **Auto-surfaced concerns** (per `_eval-template.md § Auto-surfaced
severity rules`, plus skill-specific surfaces):
- `[BLOCKER]` when `deployability` ≤ 3 — payment rate below the
plausible local market floor, or a verification threshold class
flatly unworkable for the named field context.
- `[WARN]` for a soft deployability concern (rate thin but above
floor; a threshold survivability concern that degrades but doesn't
break field use).
- `[WARN]` for each PDD Layer A rule missing from Connect verification
flags where Connect *could* enforce it.
- `[WARN]` for deliver-unit name or Entity ID composite mismatch.
- `[WARN]` for payment-unit count or type mismatch.
- `[PLATFORM]` for Layer A rules Connect cannot enforce today.
- `[DRIFT]` for each `connect-opp-setup_summary.md` ↔ live-state
discrepancy. One entry per field.
- `[INFO]` for each over-enforced rule (Connect enforces, PDD doesn't
require).
- `[INFO]` when the PDD declares no regional day-rate — note that
`deployability` graded affordability against an external benchmark
instead (PDD silence is a finding, not a skip).
9. **Defect-vs-cause discipline.** Mirrors `connect-program-setup-eval`
§ 8: state observations confidently, phrase causes tentatively. Use
`Observed: <fact>. Likely cause (unverified): <hypothesis>.` when
both are present. Verified causes (e.g., a follow-up
`connect_get_opportunity` after a delay rules out lag) may be stated
confidently with `verified by <probe>`.
## LLM-as-Judge Rubric
Calibration target on a non-degraded smoke-run Connect opp-setup:
- **Detection rate:** ≥ 80% of catalogued Connect-opp-setup issues from
`eval-calibration/known-issues.md § Connect opp setup` (ground truth
TBD — depends on real run, expect ~3–5 catalogued issues).
- **Inter-run variance:** ≤ 0.5 across 3 same-model runs.
- **Cross-model variance:** ≤ 1.0 for strong calibration.
This rubric ships at **provisional** until a real non-degraded
opp-setup run produces ground truth. Until then, it correctly emits
`incomplete` on degraded artifacts.
## Archetypes
| Archetype | What this skill does |
|---|---|
| `atomic-visit` | Default. Grades per-delivery payment, GPS-and-photo verification flags at strict defaults, single deliver-unit wiring. |
| `longitudinal-visits` | Grades **one** payment unit for the whole arc (NOT one per phase) with the sequence carried in `entity_id` and registration unpaid, plus the longitudinal clause of Layer A. Adds an **`entity_id`-grain sub-check** under `archetype_config_coherence`: the composite must resolve to entity + sequence position, and the grader checks the three failure modes `connect-opp-setup § Archetypes` enumerates — (1) the case id **alone**, which folds the whole arc onto one entity row; (2) a `concat(username, today())`-style key, which silently reverts to cross-sectional grouping; (3) **a sequence component that is present but UNCLAMPED**, which produces a byte-identical composite and puts every visit past the PDD's per-entity cap on its own key. **None of the three stops a payment** — Connect pays a repeated key again with the `duplicate` flag off (ace#2512) — so a PDD per-entity cap is graded as enforced ONLY when a `form_field_rules` row on the Deliver form's `payable_slot` field was saved; a record that cites the key alone as the cap's enforcement is a hard deduction. Mode (3) is invisible to a reader of the composite alone, so a record that merely LISTS the components is not evidence — the record must state, per component, whether the PDD's sequence/clamp semantics are implemented (ace#1462, ace#1810). |
| `focus-group` | Grades per-session payment, room-level GPS + attendance-check verification flags, FGD-form deliver-unit wiring. Adds a "facilitator-stipend vs participant-incentive" sub-check under `payment_unit_fit`. |
| `multi-stage` | Grades per-stage payment structure with stage-gate-aware verification flags. The Stage Gate from the PDD must show up in Connect as a per-stage transition rule. Adds a stage-coverage sub-check under `archetype_config_coherence`. |
## MCP Tools Used
See `skills/_eval-template.md § MCP Tools Used (stock)` for the Drive
block. Plus:
- ace-connect MCP (when `connect_opportunity_id` is real, not TBD):
`connect_get_opportunity`, `connect_list_payment_units`,
`connect_list_deliver_units`. Skip these calls in degraded mode.
## Mode Behavior
See `skills/_eval-template.md § Mode Behavior (stock)`.
## Dry-Run Behavior
Per `skills/_eval-template.md § Dry-Run Behavior (stock)`, plus skip
the live `connect_get_*` / `connect_list_*` MCP calls (read-only but
treated as offline-unsafe under `--dry-run`).
## Change Log
| Date | Change | Author |
|------|--------|--------|
| 2026-10-16 | **The four fact dimensions are graded in code (`lib/connect-opp-config-check.ts`, `scripts/fact-check-eval.ts connect-opp`); the LLM grades only `deployability` + `archetype_config_coherence`.** Verification fidelity is now what Connect can actually hold — decided `form_field_rules` vs `form_field_rules_saved`, JSONPath not XPath (ace#1301), string predicate (ace#2296), live DU id — because `gps`/`duplicate`/`gps_radius_meters` have no Connect form input and are refused (ace#1013); a PDD rule needing them is a coherence question, not a fidelity miss. Payment-unit fit diffs the create response against `payment_rate_*`, `daily_cap_per_flw`, `total_cap_per_flw`, `opportunity_currency`, with the longitudinal one-PU shape (ace#2294); activation uses `checkOppPostcondition` (invite-row proof, ace#617). Evidence: an Opus judge re-reading these payloads to grade comparisons (ACE run-efficiency audit, 2026-10-16). | ACE team |
| 2026-05-09 | Initial version. 5 dimensions: verification_flag_fidelity (0.25), payment_unit_fit (0.20), deliver_unit_wiring (0.20), active_window_status (0.20), archetype_config_coherence (0.15). Sibling of `connect-program-setup-eval`; together they cover Phase 4. Inflation guard at 8.5; same severity tier set ([BLOCKER]/[WARN]/[DRIFT]/[PLATFORM]/[INFO]/[INFO-SKIPPED]). Explicit `incomplete` verdict for degraded-mode artifacts. Live-state-drift check via `connect_get_opportunity` + `connect_list_payment_units` + `connect_list_deliver_units`. Ships at provisional calibration until a non-degraded run produces ground truth. Closes the "not yet migrated" registry row for `connect-opp-setup`. | ACE team |
| 2026-05-29 | Add out-of-chain fitness dimension per `docs/superpowers/specs/2026-05-29-eval-fitness-gap.md`. New `deployability` (0.22) grades payment-rate affordability against a *real regional day-rate benchmark* (not a PDD-declared one) and verification-threshold runtime-survivability for real FLWs — with a hard-gate (≤3 → suite `fail`) so a faithful-but-undeployable opp can't clear `pass`. Conformance dims reweighted down: verification_flag_fidelity 0.25→0.20, payment_unit_fit 0.20→0.16, deliver_unit_wiring 0.20→0.16, active_window_status 0.20→0.16, archetype_config_coherence 0.15→0.10. Weights sum to 1.00. The old `[INFO-SKIPPED]` "never deduct" payment-rate sanity sub-check (which conferred no deduction power) is retired — affordability now has teeth inside `deployability`. | ACE team |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |