Skip to content

dimagi-internal/ace / skills/connect-program-setup-eval

connect-program-setup-eval

Verified

Internal /ace:run step — Grade the run's Connect program setup against its PDD.

FileKindSize
SKILL.mdskill md23.8 KB
---
name: connect-program-setup-eval
description: >
  Internal /ace:run step — Grade the run's Connect program setup against its PDD.
user-invocable: false
disable-model-invocation: false
---

# Connect Program Setup Eval

The Connect-side configuration of an ACE opportunity is the
artifact that ties everything together: the PDD's intervention design
becomes a Program; the Learn/Deliver apps become Opportunity-linked
Connectify modules; the verification rules in the PDD's Evidence Model
become Connect's Layer A delivery-proof rules. This skill grades whether
that translation was faithful.

But faithful-to-the-PDD is not the same as fit-to-deploy. Most
dimensions here triangulate PDD ↔ Connect-config (fidelity to the AI
authoring chain). Per
`skills/_eval-template.md § The out-of-chain fitness requirement`, the
**`deployability`** dimension (0.22) is the out-of-chain fitness axis:
payment-rate affordability against a *real regional day-rate benchmark*
(not a PDD-declared one) and whether the verification thresholds are
runtime-survivable for real FLWs — with a hard-gate so a faithful-but-
undeployable program can't clear `pass`. The `program_fit_decision`
dimension is also graded out-of-chain: the rubric re-derives reuse-vs-
create *independently* rather than reading the skill's own self-reported
rationale.

Sibling rubric to `pdd-to-deliver-app-eval`, `pdd-to-learn-app-eval`,
and `idea-to-pdd-eval`. See `skills/_eval-template.md` for shared
contracts, `skills/eval-calibration/SKILL.md` for calibration
methodology, and `docs/eval-calibration-learnings.md` for patterns
and anti-patterns from the first calibrated rubrics.

## Inputs

Pipeline-only: grades the program a run's Phase 4 set up against that run's PDD.

| Source | Artifact | Used for |
|---|---|---|
| Phase 1 | `1-design/idea-to-pdd.md` | source PDD; Evidence Model + verification spec drive expectation |
| Phase 4 | `4-connect/connect-program-setup.md` and `4-connect/connect-opp-setup.md` | program + opportunity config under judgment |
| Phase 3 | `3-commcare/app-deploy_summary.md` | HQ app IDs for cross-check on Connectify wiring |

## Products

- `4-connect/connect-program-setup-eval_verdict.yaml` — verdict YAML per `_eval-template.md § Verdict YAML contract`

## Process

1. **Read inputs from GDrive** (paths in `## Inputs` above).
   Additional sources read on demand:
   - Connect setup summary: `ACE/<opp-name>/runs/<run-id>/4-connect/connect-setup_summary.md`
     (or `connect-setup/program.md` + `connect-setup/opportunity.md`).
   - Deployment summary: `ACE/<opp-name>/runs/<run-id>/3-commcare/app-deploy_summary.md` (for
     verifying the linked HQ apps match what Connect actually points
     at).

2. **Detect degraded mode.** If the connect-setup artifacts contain
   `connect_program_id: TBD-MANUAL` or `connect_opportunity_id:
   TBD-MANUAL` — i.e., Phase 4 ran in degraded mode because the
   ace-connect MCP `create_*` tools weren't yet implemented — emit a
   `verdict: incomplete` immediately with `[INFO] degraded-mode
   artifacts; not gradable as Connect-real`. **Do not score zero or
   warn for degraded mode** — it's a structural gap in the
   environment, not a quality defect in the work. The rubric is
   explicitly designed to surface degraded mode as `incomplete` rather
   than blame the operator. Once the ace-connect MCP creation tools
   land (CCC-301; ace-connect 0.8.0/0.8.1 fulfill this), real
   verdicts become possible.

3. **Extract the PDD's Connect-relevant spec.** Build a structured
   expectation:
   - Program domain (food-safety, market-survey, vaccine-hesitancy, etc.)
   - Opportunity archetype (`atomic-visit` / `longitudinal-visits` /
     `focus-group` / `multi-stage`)
   - Delivery type (data-collection, focus-group-facilitation, etc.)
   - Verification rules from Evidence Model § Layer A (GPS accuracy,
     photo presence, consent gate, market-hours window).
   - Delivery units (named slots in the Deliver app, typically the
     Connectify Deliver Unit name).
   - Payment units (per-delivery, per-day, per-piece — derive from
     Operational caps section).
   - Entity ID composite (typically declared in the Deliver app
     summary, but Connect must read the same composite for cross-
     opp duplicate detection).
   - Active window duration (Timeline section).

4. **Extract the built Connect config** from the connect-setup artifacts
   (and via `mcp__plugin_ace_ace-connect__connect_get_program` /
   `connect_get_opportunity` if the IDs are real, not TBD).

5. **Grade across 6 dimensions.** Each dimension is 0–10. Overall
   score is the weighted mean.

   | Dimension | Weight | Criteria |
   |---|---|---|
   | **Deployability (out-of-chain fitness)** | 22% | The out-of-chain anchor. Does NOT grade against the PDD — grades whether the program/opp as configured would actually work for real FLWs in the real region. **(a) Payment-rate affordability.** Estimate per-FLW daily earning (per-delivery rate × realistic max-daily-visits) and compare to a *real-world regional day-rate benchmark* for that geography and worker class (gig/CHW field rates for the country/region named in the PDD — use general domain knowledge of local market floors, NOT a PDD-declared figure). Below a plausible local market floor (a rate no real FLW would accept) = **hard-gate fail (≤3)** — the program won't recruit. Implausibly high (>3× a plausible day-rate, signalling a units/currency error) = 2-point deduction. **(b) Threshold runtime-survivability.** Would the verification thresholds wrongly reject legitimate deliveries or pass fraudulent ones in field conditions? A GPS radius too tight for real GPS jitter / tree cover / indoor markets (e.g. ≤10 m hard reject) bounces legitimate FLWs → 3-point deduction; one so loose it can't distinguish the site from a neighbour (e.g. >2 km) passes fraud → 3-point deduction; a photo/consent gate with no low-bandwidth fallback → 2-point deduction. Flatly unworkable for the named field context = **hard-gate fail (≤3)**. Anchors: deployable-as-configured = **9.0**; one soft survivability concern = **6.5**; rate below floor OR a threshold class unworkable = **≤3**. Exempt from any deferral carve-out — grade what a *deployable* program should contain even where the PDD was silent (PDD silence on a benchmark is a finding, not a free pass). |
   | **Program-fit decision** | 12% | Did Phase 4 reuse an existing Program when a clean fit existed, or create a new one when no fit existed? **Re-derive this independently — do NOT read the skill's own `connect-setup-summary.md` rationale section as the answer.** Reconstruct the correct decision from first principles: the PDD's program domain (food-safety, market-survey, vaccine-hesitancy, etc.) + the set of Programs already on the Connect org (via `connect_list_programs` when live) + the reuse criteria (same domain, same delivery type, same verification family). Then compare the skill's *action* (which Program it actually used, per the live `connect_get_program` / opp link) against your independent derivation. Reuse-when-fit and create-when-no-fit both = 10. Create-when-fit (missed reuse opportunity) = 6. Reuse-when-no-fit (forced fit, wrong domain) = 4. The summary's stated rationale is read ONLY as a `[DRIFT]` cross-check — if the skill's self-reported rationale disagrees with its own action, that's a `[DRIFT]` note, not the basis of the score. |
   | **Verification-rule fidelity** | 22% | The PDD's Evidence Model § Layer A specifies hard verification rules (GPS ≤Xm, photo present, consent=yes, market-hours window). Connect's verification flags must enforce the same rules — or, where Connect can't enforce a specific rule, the gap must be documented in `auto_surfaced`, not silently dropped. Missing a Layer A rule from Connect = 2-point deduction per rule. Adding a rule Connect enforces but the PDD doesn't require = 0.5-point deduction (over-enforcement is also a defect). |
   | **Delivery-unit wiring** | 16% | The Connect Opportunity must link to the same Deliver Unit name the Deliver app declares (Connectify-tagged form name). Mismatch is a 4-point deduction (Connect can't credit FLW visits). The Entity ID composite formula must match what the Deliver app computes for cross-opp duplicate detection. Mismatch in formula structure (e.g. PDD says `market_name + GPS hash`, Connect reads `market_name + landmark`) is a 3-point deduction. |
   | **Payment-unit fit** | 14% | Payment structure must match the PDD's intent: per-delivery for atomic-visit, **one unit for the whole arc for longitudinal-visits** (sequence in `entity_id`, registration unpaid — NOT one unit per phase), per-session for focus-group, per-stage for multi-stage. Mismatch is a 3-point deduction. **A `longitudinal-visits` opp has none of the other three payment shapes and must not be read as a mismatch against them** — that is ace#2294: this row named only the other three, so on `bednet-check-2-visit/20260908-1544` a correctly configured opportunity was gradeable as a 3-point deduction on a 14%-weighted dimension, and only survived because the judge noticed the gap and reasoned around it. For that archetype also check the **`entity_id` grain** — the composite must carry entity + sequence position, and a present-but-UNCLAMPED sequence component is byte-identical to a clamped one, so the record must state the clamp semantics per component rather than merely listing them (ace#1462, ace#1810; the three failure modes are enumerated in `connect-opp-setup § Archetypes`). (Rate *affordability* against a real-world benchmark is graded in **Deployability**, not here — this dimension is structural-fit only.) |
   | **Active-window + status** | 14% | Active-window duration matches PDD Timeline section ±10%. Status at end of Phase 4 must be `active` **with `is_test=true`** — `connect-opp-setup` Step 6.5 activates synchronously by design so Phase 6 `app-screenshot-capture` has a real signed-in test user, and the `is_test` flag excludes the opp from prod LLO-facing analytics / payment exports / partner dashboards, so this is NOT premature go-live (real-LLO go-live is Phase 9 `llo-launch`, idempotent on an already-active opp). The fail cases (≤3) are: a `draft`/never-activated opp at end of Phase 4 (Phase 6 then can't capture screenshots), OR activation **without** `is_test=true` (a real opp leaking into prod analytics before apps are tested and the bot is gated). |

   **Deduction rules:**
   - Any single dimension ≤3 → suite verdict `fail`, regardless of
     overall mean.
   - **Deployability hard-gate.** If `deployability` ≤ 3 (rate below the
     plausible local market floor, or a verification threshold class
     flatly unworkable for the named field context), the suite verdict
     is `fail` regardless of how faithfully the config matches the PDD.
     A faithful-but-undeployable program cannot clear `pass`. Emit a
     `[BLOCKER]`.
   - **Inflation guard (mirrors OCS / deliver-app / learn-app rubrics):**
     if the rubric surfaces ≥2 `[WARN]`-tier `auto_surfaced` entries,
     overall is capped at **8.5**. `[PLATFORM]` and `[DRIFT]` entries
     are tracked separately (see § Severity tiers below) and do NOT
     count toward this guard — penalizing a Phase-4 skill for a
     Connect platform limit conflates "thing the operator can fix"
     with "thing they cannot."
   - **Pre-cap and post-cap reporting** per `eval-calibration` § 0.9.4
     guidance.

   **Verdict tiers:**
   - `pass` — overall ≥ 7.0, no dimension ≤ 3, live state verified.
   - `partial` — overall ≥ 7.0, no dimension ≤ 3, **artifact looks
     correct on paper but live MCP probes failed at grading time**
     (network, auth, transient Connect 5xx). Set `live_state_verified:
     false`. Caps overall at **8.5** to mark not-fully-verified.
     Different from `incomplete` (which is reserved for structural
     gaps in the artifact itself, e.g. degraded-mode `TBD-MANUAL`
     IDs). `partial` says "score is from artifact text alone — re-run
     when MCP is reachable."
   - `warn` — overall ≥ 5.0 < 7.0, or any inflation cap binds.
   - `fail` — overall < 5.0 OR any dimension ≤ 3.
   - `incomplete` — degraded-mode `TBD-MANUAL` artifacts, missing
     PDD or connect-setup-summary, or any other structural gap that
     makes grading impossible.

   **Severity tiers** for `auto_surfaced` entries:
   - `[BLOCKER]` — must-fix before merge. Counts as a hard defect.
   - `[WARN]` — should-fix; counts toward inflation guard.
   - `[DRIFT]` — `connect-setup-summary.md` claims disagree with
     `connect_get_*` live read. Diagnostic-only; the rubric records
     the discrepancy but does NOT deduct (the *score* dimension that
     consumes either source already deducts if either side is wrong;
     deducting a second time for the disagreement double-counts).
   - `[PLATFORM]` — defect originates in Connect itself, not the
     skill's output. Examples: a Connect form field that doesn't
     accept the spec'd value, an HTMX endpoint that returns no
     citation markup, an unsupported verification rule. Does NOT
     count toward inflation guard.
   - `[INFO]` — observational, no action required.

6. **Write the verdict YAML** to
   `4-connect/connect-program-setup-eval_verdict.yaml` using the shape
   from `skills/_eval-template.md § Verdict YAML contract`. Dimensions:

   ```yaml
   dimensions:
     deployability:               { weight: 0.22 }   # out-of-chain fitness; hard-gate at ≤3
     program_fit_decision:        { weight: 0.12 }   # independently re-derived, not read from skill rationale
     verification_rule_fidelity:  { weight: 0.22 }
     delivery_unit_wiring:        { weight: 0.16 }
     payment_unit_fit:            { weight: 0.14 }
     active_window_status:        { weight: 0.14 }
   # weights sum to 1.00
   ```

   Always set `live_state_verified` based on whether `connect_get_*`
   probes succeeded — false forces verdict ≤ partial.

7. **Auto-surfaced concerns** (per `_eval-template.md § Auto-surfaced
   severity rules`, plus skill-specific surfaces):
   - `[BLOCKER]` when `deployability` ≤ 3 — payment rate below the
     plausible local market floor, or a verification threshold class
     flatly unworkable for the named field context.
   - `[WARN]` for a soft deployability concern (rate thin but above
     floor; a threshold survivability concern that degrades but doesn't
     break field use).
   - `[WARN]` for each PDD Layer A rule missing from Connect verification
     where Connect *could* enforce it.
   - `[WARN]` for Delivery Unit name or Entity ID composite mismatch.
   - `[PLATFORM]` for Layer A rules Connect cannot enforce today
     (e.g. district-specific market-hours window). These document the
     gap without penalizing the skill.
   - `[DRIFT]` for each `connect-setup-summary.md` ↔ `connect_get_*`
     discrepancy. One entry per field. Also `[DRIFT]` when the skill's
     self-reported program-fit rationale disagrees with its own action
     (the score comes from the independent re-derivation, not the
     rationale text).
   - `[INFO]` for each over-enforced rule (Connect enforces, PDD doesn't require).
   - `[INFO]` when the PDD declares no regional day-rate — note that
     `deployability` graded affordability against an external benchmark
     instead (PDD silence is a finding, not a skip).

   **Live-state-drift check (new in 0.10.7):** when `live_state_verified
   = true`, after grading the dimensions from `connect-setup-summary.md`,
   compare each summary claim against the live `connect_get_program` /
   `connect_get_opportunity` / `connect_list_payment_units` /
   `connect_list_deliver_units` response. Emit a `[DRIFT]` entry per
   discrepancy. Drift is diagnostic, not deductive — the dimension
   score already reflects whichever source is wrong; counting it twice
   double-penalizes. The drift log is the audit trail that lets
   downstream investigation know which side to trust.

8. **Defect-vs-cause discipline.** When writing `auto_surfaced` and
   `per_item.note` text:
   - **State the observation confidently.** "X was created with all
     fields filled but `connect_get_program` returns empty fields" is
     a fact you saw.
   - **Phrase causes tentatively** — "consistent with", "one possible
     cause is", "the symptom matches a serialization or read-path
     gap." Never assert "this is a serialization gap" unless you
     traced it. LLM-as-Judge tends to pattern-match to the most
     familiar root cause; that's frequently wrong.
   - When the message includes both, separate them:
     `Observed: <fact>. Likely cause (unverified): <hypothesis>.`
   - Verified causes (where the rubric ran a probe, e.g. a follow-up
     `connect_get_program` after a delay to rule out lag) may be
     stated confidently — note "verified by <probe>".

   This rule was added 0.10.6 after `connect-program-setup-eval` on
   `turmeric-market-survey-2026-04-28` correctly flagged a real bug
   (Program edit form fields missing post-create) but mis-attributed
   it to a write-side serialization gap — the actual cause was a
   read-side hydration bug in `getProgram`. The eval was right about
   the defect, wrong about the layer.

## LLM-as-Judge Rubric

Calibration target on a non-degraded smoke-run Connect setup:

- **Detection rate:** ≥ 80% of catalogued Connect-setup issues from
  `eval-calibration/known-issues.md § Connect setup` (ground truth
  TBD — depends on real run, expect ~3–5 catalogued issues).
- **Inter-run variance:** ≤ 0.5 across 3 same-model runs.
- **Cross-model variance:** ≤ 1.0 across Sonnet/Opus/Haiku for strong
  calibration.

This rubric ships at **provisional** until a real non-degraded
Connect run produces ground truth. Until then, it correctly emits
`incomplete` on degraded artifacts and provides the framework for
future grading.

## Archetypes

| Archetype | What this skill does |
|---|---|
| `atomic-visit` | Default. Grades per-delivery payment, GPS-and-photo verification rules, single-Deliver-Unit wiring. |
| `longitudinal-visits` | Grades **one** payment unit for the whole arc — not one per phase — with the sequence expressed in `entity_id` and registration unpaid, plus the longitudinal clause of Layer A. Adds an **`entity_id`-grain sub-check** under `payment_unit_fit`: the composite must resolve to entity + sequence position, against the three failure modes `connect-opp-setup § Archetypes` enumerates — (1) the case id **alone** folds the whole arc onto one entity row; (2) a `concat(username, today())`-style key reverts to cross-sectional grouping; (3) **the sequence component is present but UNCLAMPED**, so the PDD's per-entity grouping silently does not hold. None of the three stops a payment — a repeated key is paid again (ace#2512) — so the per-entity cap is graded as enforced only with a saved `payable_slot` `form_field_rules` row. Mode (3) produces a byte-identical composite, so a record that lists the components is not evidence — it must state, per component, whether the clamp is implemented (ace#1462, ace#1810). |
| `focus-group` | Grades per-session payment, attendance + per-domain-summary verification rules, FGD-form Delivery-Unit wiring. Adds a "facilitator-stipend vs participant-incentive" sub-check under payment_unit_fit. |
| `multi-stage` | Grades per-stage payment structure with stage-gate-aware verification rules. The Stage Gate from the PDD must show up in Connect as a status transition rule. |

## MCP Tools Used

See `skills/_eval-template.md § MCP Tools Used (stock)` for the Drive
block. Plus:
- ace-connect MCP (when `connect_program_id` and `connect_opportunity_id`
  are real, not TBD): `connect_get_program`, `connect_get_opportunity`,
  `connect_list_payment_units`, `connect_list_deliver_units`,
  `connect_list_programs` (the last enumerates existing org Programs so
  `program_fit_decision` can be re-derived independently of the skill's
  own rationale). These
  let the rubric verify the live Connect state against the
  `connect-setup-summary.md` claims, catching skill-output-vs-actual
  drift. Skip these calls in degraded mode.

## Mode Behavior

See `skills/_eval-template.md § Mode Behavior (stock)`.

## Dry-Run Behavior

Per `skills/_eval-template.md § Dry-Run Behavior (stock)`, plus skip
the live `connect_get_*` MCP calls (read-only but treated as offline-
unsafe under `--dry-run`).

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-04-28 | Initial version. 5 dimensions: program_fit_decision (0.15), verification_rule_fidelity (0.25 — most load-bearing for Layer A faithfulness), delivery_unit_wiring (0.20), payment_unit_fit (0.20), active_window_status (0.20). Inflation guard at 8.5. Explicit `incomplete` verdict for degraded-mode artifacts (the Phase 4 mode that ran on smoke-20260428-1242 before ace-connect MCP shipped) — degraded mode is environment, not quality, and shouldn't deduct. Ships at provisional calibration until a non-degraded run produces ground truth. | ACE team (eval system buildout — 0.9.8) |
| 2026-04-29 | Defect-vs-cause discipline added (step 8). Driven by the `turmeric-market-survey-2026-04-28` run mis-attributing a real read-side hydration bug to a write-side serialization gap. The rubric correctly flagged the symptom; the rubric did not constrain causal attribution. Now requires confident observation + tentative cause, with `Observed: ... Likely cause (unverified): ...` formatting when both are present. | ACE team (0.10.6) |
| 2026-05-29 | Add out-of-chain fitness dimension per `docs/superpowers/specs/2026-05-29-eval-fitness-gap.md`. New `deployability` (0.22) grades payment-rate affordability against a *real regional day-rate benchmark* (not a PDD-declared one) and verification-threshold runtime-survivability for real FLWs — hard-gate (≤3 → suite `fail`) so a faithful-but-undeployable program can't clear `pass`. `program_fit_decision` is now graded by **independently re-deriving** reuse-vs-create (PDD domain + `connect_list_programs` + reuse criteria) rather than reading the skill's self-reported rationale; the rationale is now only a `[DRIFT]` cross-check. Reweighted to fit `deployability`: program_fit_decision 0.15→0.12, verification_rule_fidelity 0.25→0.22, delivery_unit_wiring 0.20→0.16, payment_unit_fit 0.20→0.14, active_window_status 0.20→0.14. Weights sum to 1.00. Retired the `[INFO-SKIPPED]` "never deduct" payment-rate sanity sub-check (no deduction power) — affordability now has teeth inside `deployability`. | ACE team |
| 2026-04-29 | Five-item rubric polish from the turmeric run's first non-degraded grading. (1) Added `partial` verdict tier for runtime-blocked-but-not-degraded mode (artifact correct, live MCP probes unreachable). Caps at 8.5; `live_state_verified: false`. (2) `[PLATFORM]` severity tier for defects that originate in Connect itself rather than skill output; does NOT count toward the inflation guard. Removes a class of false-deduction where the skill is penalized for a Connect schema limit. (3) `[DRIFT]` severity tier for `connect-setup-summary` ↔ live-state discrepancies; diagnostic-only, never deductive (the dimension consuming either source already deducts if either is wrong; counting drift again double-penalizes). New live-state-drift check runs after dimensional grading. (4) Payment threshold-sanity sub-check now explicitly conditional: if PDD declares no regional day-rate, emit `[INFO-SKIPPED]` and skip — do NOT count as defect. Documents the coverage gap without penalizing. (5) `live_state_verified` boolean added to verdict schema; forces verdict ≤ `partial` when false. | ACE team (0.10.7) |
| 2026-09-06 | **Stop routing concerns to a gate brief that does not exist (dimagi-internal/ace#1884).** 0.13.116 removed the per-skill gate-brief file class and the ace#1880 sweep removed the remaining `*.md` PATHS, but prose directives naming the gate brief as a DESTINATION survived in 15 files — a concern "surfaced in the gate brief" is surfaced nowhere. Repointed at the verdict YAML's `auto_surfaced` block, which is what the orchestrator actually renders the pause summary from. Gated by the new destination check in `test/skills/gate-brief-removal-complete.test.ts`. | ACE team |
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall