dimagi-internal/ace / skills/ocs-widget-handoff-eval
ocs-widget-handoff-eval
VerifiedInternal /ace:run step — Grade the run's OCS widget handoff before Connect.
OCS Widget Handoff Eval
The OCS chatbot is staged at the end of Phase 5 (ocs-setup) so the LLO can
reach it. Connect has no per-opportunity widget field today (CCC-301), so
there is nothing to paste it into — connect_update_opportunity carries no
widget parameter and mcp/connect/ contains no widget or embed surface at all.
The route that works right now is the bot's public link
(https://www.openchatstudio.com/c/<token>/, OCS's supported anonymous chat
page), which needs no configuration hop. Runs before 2026-10-16 carry the legacy
/start/ page instead, which OCS is removing (dimagi/open-chat-studio#4209).
This rubric grades the staging artifact (5-ocs/ocs-setup_widget-handoff.md) on
whether it gives the operator what they can actually use today: a live public
chat URL, the credentials held for when CCC-301 ships, the opportunity this bot
is bound to, and a way to verify they are talking to the right bot.
See skills/_eval-template.md for shared contracts. Authored 0.10.29 to
absorb turmeric run_time_followups item 10 (HITL widget paste-in until
CCC-301).
Inputs
Pipeline-only: grades a run's Phase 5 widget handoff, which exists only in a run.
| Source | Artifact | Used for |
|---|---|---|
| Phase 5 | 5-ocs/ocs-setup_widget-handoff.md | staging artifact under judgment |
| Phase 5 | 5-ocs/ocs-agent-setup.md | cross-check embed_key, experiment_id, public_id |
| Per-run | runs/<run-id>/run_state.yaml | connect_opportunity.url |
Products
5-ocs/ocs-widget-handoff-eval_verdict.yaml— verdict YAML per_eval-template.md § Verdict YAML contract
Process
This rubric is graded in CODE (2026-10-16). URL shape + live probe,
opportunity binding, credential hygiene and the three instruction elements
are all checks on text and one HTTP response, so
lib/ocs-widget-handoff-check.ts computes them — including the anonymous
probe of the public link and a scan for every configured secret VALUE in the
plugin .env, which no reader can do by eye. The ONLY judgment left is an
optional readability deduction (0–2 points, with a reason) on
operator_instructions_clarity. The steps after this block are the spec the
code implements; do not re-grade them by hand.
-
drive_read_file→5-ocs/ocs-setup_widget-handoff.mdand the run'srun_state.yaml; save both locally. -
Run (add
--no-probeunder--dry-run):ACE_ROOT="${CLAUDE_PLUGIN_ROOT:-$(python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json'))); print(d['plugins']['ace@ace'][0]['installPath'])")}" node "$ACE_ROOT/node_modules/tsx/dist/cli.mjs" "$ACE_ROOT/scripts/fact-check-eval.ts" ocs-widget-handoff \ --handoff <handoff.md> --run-state <run_state.yaml> --target <opp>/<run-id> --out <local verdict.yaml> -
Optional: if the handoff would confuse a non-technical reader in a way the element checks cannot see, re-run with
--clarity-residue <json>({"deduction": 0..2, "note": "<why>"}). -
drive_create_filethe--outfile to5-ocs/ocs-widget-handoff-eval_verdict.yaml. -
Read inputs from GDrive (paths in
## Inputsabove). -
Detect missing artifacts. If
5-ocs/ocs-setup_widget-handoff.mdis missing, emitverdict: incompletewith[INFO] widget-handoff.md not found — Phase 5 ocs-setup did not write the staging artifact, or the LLO already pasted in and the artifact was archived. -
Grade across 4 dimensions.
Dimension Weight Criteria Widget URL resolves 25% The handoff must carry the bot's public link — https://www.openchatstudio.com/c/<link_token>/, exactly thepublic_urlocs_ensure_public_linkreturned inocs-agent-setupStep 10 — plus thepublic_idand a non-emptyembed_keyfor the widget element. Probe it anonymously and require a 200 with neither banner (This chatbot is not published yet./This chatbot is temporarily unavailable.); no cookie jar is needed for this page. A handoff that still carries only the legacy/a/<team_slug>/chatbots/<public_id>/start/URL = ≤6: OCS is removing that page (dimagi/open-chat-studio#4209, #4792), so the link works today and stops without warning. The rest of this cell describes that legacy page, for grading runs recorded before 2026-10-16 — for those, probe it anonymously and require a 200. The probe MUST carry cookies through the redirect (curl -L -c jar -b jar, or a Playwright context):start_session_publiccreates a session and 302s to/s/<session_id>/chat/, which is wrapped in@verify_session_access_cookie, so a cookieless fetch reads 404 on a perfectly working bot. Verified live 2026-08-14 againstconnect-ace/f92d26f3-…: no jar → 302 then 404; with jar → 200. URL-shape-correct but genuinely unreachable = ≤6 ([PLATFORM]). A503carrying OCS's maintenance page is[PLATFORM], never a handoff defect —start_session_publicreturns it when the team'splatform=WEBExperimentChannelis disabled (OCS #4230), a team-wide admin kill-switch that takes down every ACE per-opp chat URL at once and says nothing about this artifact. Score the URL on its shape and note the cause; do not deduct (ace#1812). URL-shape-broken, or the retired/chatbots/embed/<public_id>/path (a 410 stub since OCS #3540, 2026-08-03) = ≤3. Do NOT deduct for the absence of an embed page — there isn't one, and the handoff naming thestart/route instead is correct, not a defect (ace#1021).Connect opp binding 20% widget-handoff.md must name the Connect opportunity this chatbot is bound to, so a reader can tell which opp's bot they hold — an opp typically has several runs' bots and they look alike. This is an IDENTITY reference, not a paste target: there is no per-opportunity widget field to paste into (see the header). Mismatch with run_state.yaml's connect_opportunity.url= 4-point deduction. Missing = ≤4.Operator instructions clarity 30% The handoff must tell a non-technical reader (a) what to do today — share the public chat URL, which requires no configuration step — (b) how to verify they have the right bot — a chat-test prompt whose answer is opp-specific confirms the OPPORTUNITY, and the public_idin the link confirms the RUN (every run of an opp has its own bot built from the same PDD, and the public chat page shows neither bot name nor opportunity, so a handoff that names the bot's run-id or the opportunity as the run check points at something the reader cannot see — a 1-point deduction within this element; ace#2488), and (c) what thepublic_id/embed_keyare for, namely the embed that becomes possible when CCC-301 ships. Each missing element = 2-point deduction. Requiring a Connect paste-in is an AUTO-FAIL for this dimension, not a bonus (≤3): an instruction to find a "support chatbot / help widget field group" sends a reader hunting for a UI element that does not exist, and four sibling skills —training-faq,training-quick-reference,training-flw-guide,_training-template— state so verbatim. A handoff that says plainly no paste-in is possible yet is CORRECT and scores full marks (ace#1811).Credential hygiene 25% embed_keyis opp-specific (an OCS bot-level value), NOT a global API key — its presence is expected and is never a finding. The guard protects against exactly one thing: a credential that grants access beyond this single opportunity appearing in an LLO-facing artifact —OCS_API_TOKEN,OCS_PASSWORD,OCS_GOLDEN_TEMPLATE_ID,LABS_MCP_TOKEN, or any HQ / Connect credential. Surface a[WARN]per such value; a leak is the auto-fail below.link_token,team_slugandpublic_idare public-by-construction and are NEVER a hygiene finding. The public linkwidget_url_resolvesREQUIRES (row above) is/c/<link_token>/, and that token is the public-link channel's own embed key — OCS prints it into the page it serves to every visitor, so it grants nothing beyond chatting with this one bot. The legacy/start/route is team-scoped, so it necessarily contains the team slug. Deducting for either would fail a correct handoff on the strength of the dimension that mandates it (ace#1680).Deduction rules:
- Any single dimension ≤3 → suite verdict
fail. - Inflation guard: ≥2
[WARN]→ cap at 8.5. [PLATFORM]and[DRIFT]do NOT count.
Verdict tiers:
pass— overall ≥ 7.0, no dimension ≤ 3, widget URL fetched 200.partial— overall ≥ 7.0 but the widget HTTP probe failed at grading time. Cap 8.5;live_state_verified: false.warn— overall ≥ 5.0 < 7.0 OR inflation cap binds.fail— overall < 5.0 OR any dimension ≤ 3 OR a credential that grants access beyond this opportunity leaked into the handoff (auto-fail; security guard).team_slugandpublic_idare not credentials — see thecredential_hygienerow.incomplete— widget-handoff.md missing.
- Any single dimension ≤3 → suite verdict
-
Severity tiers:
[BLOCKER]for any dimension ≤ 3, missing Connect opp link, or credential leak (a cross-opportunity credential in an LLO-facing doc). Never forteam_slug/public_id.[BLOCKER]if overall < 7.0.[WARN]per missing operator-instruction element.[PLATFORM]for the OCS public chat route returning 5xx during the probe (handoff structurally correct; live verification unavailable). A 404 with no cookie jar is not a platform issue — it is a defective probe; re-run it with cookies before recording anything (ace#1021).[PLATFORM]for Connect's lack ofupdate_opportunitywidget-config API (the entire reason this hop is HITL today). One canonical entry per verdict; ties to CCC-301.[DRIFT]per widget-handoff claim ↔ live ocs_get_chatbot or run_state.yaml disagreement.[INFO]for "public chat URL fetched OK; the LLO can be sent the link as-is."[INFO-SKIPPED]for the live HTTP probe when offline-mode is requested.
-
Write the verdict YAML to
ACE/<opp-name>/runs/<run-id>/5-ocs/ocs-widget-handoff-eval_verdict.yaml. The filename uses this eval skill's name (ocs-widget-handoff-eval), not the producer's (ocs-agent-setup— the skill that produces5-ocs/ocs-setup_widget-handoff.mdas one of its outputs): per the 0.12.0 Option-α rule an-evalskill keeps-evalin its verdict filename, and the Workbench rolls the score up to the producer row via theeval_skill:pairing in the phase agent's frontmatter, not by parsing the filename — seeagents/ace-orchestrator.md § Per-Step Eval Hook. Body conforms tolib/verdict-schema.ts.skill: ocs-widget-handoff-eval target: <opp-name> mode: deep ran_at: <ISO timestamp> capture_path: 5-ocs/ocs-setup_widget-handoff.md overall_score: 8.7 overall_score_pre_cap: 8.7 verdict: pass live_state_verified: true dimensions: widget_url_resolves: { score: 9.0, weight: 0.25 } connect_opp_binding: { score: 9.5, weight: 0.20 } operator_instructions_clarity: { score: 8.0, weight: 0.30 } credential_hygiene: { score: 9.5, weight: 0.25 } per_item: - ref: "Widget URL" score: 9.0 verdict: pass note: "Includes public_id 1fcddd08-...; embed_key wDwe70vquTL...; HTTP probe returned 200." - ref: "Connect opp binding" score: 9.5 verdict: pass note: "Names the bound opportunity and matches run_state.yaml connect_opportunity.url for 249ad8fe-...; carries it as an identity reference and correctly states no paste-in target exists yet." auto_surfaced: - severity: PLATFORM message: "No Connect paste-in exists yet: connect_update_opportunity carries no widget parameter (CCC-301), so the public chat URL is the working route. Rubric grades staging, not a paste-in." - severity: INFO message: "embed_key is opp-specific (OCS bot-level), not a cross-opportunity credential. Safe to surface in LLO-facing doc. team_slug/public_id are public-by-construction (they are in the mandated chat URL)." gate: threshold: 7.5 disposition: approve | reject | iterate
LLM-as-Judge Rubric
Provisional until 3 real handoffs produce ground truth.
Calibration target:
- Detection rate: ≥ 80% of catalogued widget-handoff issues from
eval-calibration/known-issues.md § Widget handoff(TBD). - Inter-run variance: ≤ 0.5 across 3 same-model runs.
- Cross-model variance: ≤ 1.0 for strong calibration.
The 3 calibration runs should include:
- A clean handoff (canonical pass).
- A handoff missing the Connect opp link (drives BLOCKER).
- A handoff with a leaked cross-opportunity credential — e.g.
OCS_API_TOKEN(drives auto-fail).
Archetypes
Archetype-agnostic. The widget handoff shape is identical for every archetype, so this section names none of them — an archetype-agnostic skill that lists the vocabulary anyway acquires a maintenance obligation it has no use for, and this one had gone stale (it named the pre-2026-08-17 three; ace#2312).
MCP Tools Used
- Google Drive:
drive_read_file,drive_create_file - ace-ocs MCP:
ocs_get_chatbot_embed_infoto cross-check embed_key and widget URL shape against the live bot. - HTTP probe (no MCP): GET widget URL with a HEAD request to verify the endpoint returns 200 + sensible content-type.
Mode Behavior
- Auto: Grade, write verdict.
- Review: Pause after grading.
Dry-Run Behavior
- Read inputs normally.
- Skip the widget HTTP probe and Connect opp-link probe.
- Write verdict.
- State tracks as
dry-run-success.
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-10-16 | Graded in code: lib/ocs-widget-handoff-check.ts via scripts/fact-check-eval.ts ocs-widget-handoff. Every dimension is a check on the handoff text plus one anonymous HTTP response, so an Opus judge added minutes and variance and no judgment (ACE run-efficiency audit, 2026-10-16). The code holds the hard-won rules as tests: public-by-construction values never a hygiene finding (ace#1680), 503 maintenance is PLATFORM (ace#1812), phantom paste-in auto-fails (ace#1811), legacy /start/ ≤ 6 (OCS #4209), public_id as the run check (ace#2488). Credential hygiene now also matches configured secret VALUES verbatim, not just secret-shaped strings. The only judgment left is an optional ≤ 2-point readability deduction. | ACE team |
| 2026-09-26 | Element (b) now separates the opportunity check from the run check (ace#2488). The phase contract said the opportunity reference lets a reader "tell which run's bot they hold", and (b) asked only for an opp-specific chat-test prompt. Neither discriminates runs: every run of an opp builds its own bot from the same PDD (spark-facilitator carried 7 prior-run bots at 20260925-1536), so they all answer the prompt identically, and the public chat page renders neither the bot name nor the opportunity. The public_id in the link is the only run-level check the reader can see; (b) now asks the handoff to name it. Raised as a WARN by the independent grader on spark-facilitator/20260925-1536. | ACE team |
| 2026-08-29 | The rubric required an operator instruction that does not exist, and thereby caused the defect it should have caught (ace#1811). operator_instructions_clarity (30%) mandated telling the LLO "where to paste (Connect opp config tab, specific field name)", and the "Connect opp link" dimension called the opportunity URL the one the LLO "needs to paste INTO". There is no such field: connect_update_opportunity carries no widget parameter and mcp/connect/ contains no widget or embed surface at all, while four sibling skills state "Connect has no per-opp widget field" verbatim. A handoff that told the truth therefore LOST points, and the one that shipped on hh-poverty-targeting/20260828-0702 sent an operator hunting for a UI element three files in the same repo say is absent. Same shape as the #1680 and #1021 rows below — a rubric demanding something the platform does not have — except this one propagated into an LLO-facing artifact. The dimension now grades what the reader can actually do today (share the public chat URL, verify the bot, hold the credentials for CCC-301) and makes requiring a paste-in an auto-fail. Enforced: test/skills/widget-handoff-no-phantom-paste-target.test.ts. | ACE team |
| 2026-08-29 | widget_url_resolves can now fail for a reason that is not the handoff's fault (ace#1812). OCS #4230 added a team-wide admin kill-switch: if the team's platform=WEB ExperimentChannel is disabled, start_session_public returns a 503 maintenance page for EVERY bot on the team. The rubric required a 200 and had no vocabulary for this, so a correct handoff would have been scored as a broken one during any OCS maintenance window — the same shape as the #1680 and #1021 rows below. The dimension now names a 503 maintenance page as [PLATFORM] (which does not count against the score) rather than a handoff defect. Weights unchanged. | ACE team |
| 2026-08-26 | credential_hygiene no longer contradicts widget_url_resolves (ace#1680). The dimension listed OCS_TEAM_SLUG as a "global secret" whose presence is an auto-fail security guard, while widget_url_resolves REQUIRES https://www.openchatstudio.com/a/<team_slug>/chatbots/<public_id>/start/ — a team-scoped route, so the slug is structurally unavoidable in the one artifact the rubric mandates. A correct handoff was therefore reachable, on a literal reading, as a fail on a security guard; whether it failed depended on which way the judge leaned. The guard now names what it actually protects — a credential granting access beyond this opportunity (OCS_API_TOKEN, OCS_PASSWORD, OCS_GOLDEN_TEMPLATE_ID, LABS_MCP_TOKEN, HQ/Connect creds) — and states that team_slug / public_id are public-by-construction and never a finding. Weights unchanged. Enforced: test/skills/credential-hygiene-vs-widget-url.test.ts. | ACE team |
| 2026-08-14 | widget_url_resolves probes the route that exists (ace#1021). The dimension gated 25% of the score on fetching an OCS "widget endpoint", while ocs-agent-setup already recorded that /chatbots/embed/<public_id>/ is a 404 — so a CORRECT handoff scored as badly as a broken one. Both skills were wrong about the platform: the anonymous route is team-scoped, /a/<team_slug>/chatbots/<public_id>/start/ (apps/chatbots/urls.py:80 mounted under config/urls.py:88), and it works. The embed flow really was deleted (OCS #3540, 2026-08-03). The probe now targets the real URL and MUST carry cookies through the 302 — @verify_session_access_cookie on the chat view makes a cookieless fetch read 404 on a working bot, which is the false negative that produced this issue. | ACE team |
| 2026-04-29 | Initial version. 4 dimensions: widget_url_resolves (0.25), connect_opp_link (0.20), operator_instructions_clarity (0.30 — heaviest because LLOs are non-technical), credential_hygiene (0.25 — security guard with auto-fail on global-secret leak). Provisional calibration. Absorbs turmeric run_time_followups item 10 (HITL widget paste-in until CCC-301). | ACE team (0.10.29) |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |