Skip to content

dimagi-internal/ace / skills/interview-opp-verify

interview-opp-verify

Unverified

Read-only verifier for a configured Connect Interviews opportunity. Walks docs/connect-interviews/checklist-schema.yaml, calls the read atoms, and grades each rule pass / fail / unverifiable. Cross-system consistency checks (e.g. OCS custom action target URL == HQ Inbound API URL) are included.

Interview Opportunity Verify

Walk every rule in docs/connect-interviews/checklist-schema.yaml against a live opp and produce a verdict report.

Slash command

/ace:interview-opp-verify <connect-opp-url>

Or with explicit args:

/ace:interview-opp-verify --opp <opp_id> --org <pm_org> --domain <hq_domain> --bot <experiment_id>

Input

Resolved from the Connect opp URL: /a/<org>/opportunity/<opp_id>/. Skill fetches the opp, then derives:

  • HQ downstream domain (from opp.learn_app.domain)
  • OCS bot id (from connections + bot list — defaults to env OCS_INTERVIEWS_TEMPLATE_ID if there's only one)

Products

  • ACE/<opp-name>/verify/<run-id>/report.md — human-readable verdict report
  • ACE/<opp-name>/verify/<run-id>/verdict.yaml — machine-readable per-rule pass/fail
  • Exit code 0 if all green, 1 if any fail, 2 if any unverifiable items exist

Process

The skill loads docs/connect-interviews/checklist-schema.yaml (in-repo) and iterates every item, calling the named verify.atom with the appropriate args.

Sections walked (in order):

  1. per_program — Connect program + LLO org membership + OCS team
  2. per_domain — all the HQ plumbing + the OCS bot's structural pieces
  3. per_cohort — apps, lookup rows, opp config, payment unit, conditional alert, OCS interview nodes
  4. per_user — cohort_id is set on this cohort's FLWs (V1: only checks the opp's invited users)

Verdict shapes:

  • ✅ pass — atom returned matching state per the rule
  • ❌ fail — atom returned and state mismatched
  • ⚠️ unverifiable — atom doesn't exist yet (gap) OR returned an error we can't classify
  • ⊘ out_of_band — rule explicitly marked as "humans must attest" (e.g. subscription tier)

Cross-system checks

Some rules check consistency across systems. The verifier resolves IDs cohesively before grading:

  • ocs-bot-has-completion-action.target_url_matches: "@inbound-api-session-completion.url" — verifier reads the HQ Inbound API URL, then checks the OCS custom action's server_url + api_schema operationId path resolves to the same URL.
  • opp-learn-app-linked.value: "@app-learn-copied.app_id" — verifier reads the new linked-app id from the cohort spec, checks the opp's learn_app.cc_app_id matches.

OCS rules — all read from one ocs_inspect_chatbot call

As of OCS PR #3536 (deployed 2026-06-09; refactored on FK relations in #3645 + #3652 on 2026-06-18), every OCS-side rule reads from a single ocs_inspect_chatbot({ public_id }) call. The response shape is the OpenAPI ChatbotInspect schema served at https://www.openchatstudio.com/api/schema/ — grep there for the field contract before paraphrasing. The mappings:

RuleInspect payload path
ocs-bot-has-router-nodepipeline.nodes[].type === "StaticRouterNode"
ocs-bot-has-llm-nodepipeline.nodes[].type === "LLMResponseWithPrompt" (also LLMResponse)
ocs-bot-has-python-node-session-statepipeline.nodes[].type === "CodeNode"
ocs-bot-has-24hr-timeout-eventevents.timeout_triggers[*].delay_seconds === 86400 ← experiment-level, NOT a pipeline node (this gap was the original driver for OCS #3458)
ocs-bot-has-completion-actionpipeline.nodes[].custom_actions[*].name matches "Session Completion"; server_url cross-checked vs HQ Inbound API URL
ocs-bot-has-24hr-expiry-actionpipeline.nodes[].custom_actions[*].name matches "Expiry"
ocs-interview-nodes-per-cohortrouter node params.keywords[] includes every interview_id in cohort.schedule
ocs-interview-source-materialpipeline.nodes[].source_material + indexed_collections[]

Call ocs_get_me first when running on a new machine: if team.slug doesn't match the team that owns the bot, inspect will 404 (looks like "bot doesn't exist" but is really "wrong team key").

Multi-team — target a team other than OCS_TEAM_SLUG

The three observation atoms ocs_get_me, ocs_list_chatbots, and ocs_inspect_chatbot accept an optional team_slug argument. When supplied AND it differs from the configured default team (OCS_TEAM_SLUG), the MCP server looks up the matching token from OCS_API_TOKEN_<SLUG> env vars and uses it for that call only. Default-team calls (no team_slug) keep using OCS_API_TOKEN exactly as before — no behavior change for single-tenant workflows.

To verify a bot on a non-default team:

  1. Mint a read-only UserAPIKey scoped to the target team via /users/profile/ → API keys (uncheck "allow write").
  2. Store it in ACE's Canopy vault on canopy-web (Settings → Credentials) as OCS_API_TOKEN_<SLUG_UPPER> (slug uppercased, non-alphanumeric → _).
  3. Run bash bin/ace-setup --force-env to rewrite .env from the vault.
  4. Restart Claude Code so the MCP subprocess picks up the new env var.
  5. Pass team_slug: "<slug>" on every OCS call in the verifier run.

A team_slug for which no token is registered surfaces a typed error naming the exact env var to add — so misconfigurations don't bleed into 401 mysteries.

Report format

# Interview Opp Verify — <cohort_id>

**Generated:** 2026-05-22 (run-id 20260522-0030)
**Opportunity:** https://connect.dimagi.com/a/<org>/opportunity/<opp_id>/
**HQ domain:** <slug>  **OCS bot:** <experiment_id>

## Summary

| Pass | Fail | Unverifiable | Out-of-band | Total |
|---|---|---|---|---|
| 38 | 2 | 5 | 3 | 48 |

## Per-section

### per_program (3 items)
✅ connect-program-exists — "ACE Interviews Test" found
⊘ llo-orgs-accepted-into-program — out_of_band (ACE-owned program has no external LLOs)
⊘ ocs-team-for-program — out_of_band (OCS_TEAM_SLUG=connect-ace confirmed by env)

### per_domain (24 items)
...

### per_cohort (25 items)
...

## Action items

For each fail:
- **conditional-alert-payment** — alert "<cohort_id> Payment Conditional Alert" not found on /a/<domain>/messaging/conditional/. Create manually (atom not yet built).

For each unverifiable:
- **ocs-interview-nodes-per-cohort** — atom gap (ocs_edit_pipeline_structure). Manually inspect the bot at <bot-url> and confirm each interview_id in the cohort's schedule has a router target.

Process detail

  1. Resolve identifiers. Given the opp URL, call connect_get_opportunity to fetch the opp; extract learn_app.domain as the downstream HQ domain. Works at viewer tier — ACE does not need write access on the org (ace#1461). On a viewer-tier read the opp is hydrated from the read-only dashboard instead of the edit form, so short_description and country come back EMPTY (they are not rendered there). Do not treat an empty short_description as a configuration defect on this path; re-read with member tier if you actually need those two fields.

  2. Load the schema. Read docs/connect-interviews/checklist-schema.yaml from the plugin install dir; iterate per_program / per_domain / per_cohort / per_user items.

  3. Per item:

    • If verify.atom is null, mark unverifiable with the item's notes.
    • If verify.rule.type is out_of_band, mark ⊘.
    • Otherwise, call the atom with the rule's args (substituting @<other-item-id>.field references resolved from previously-graded items).
    • Apply the rule type: exists, name_matches, property_equals, filter_matches, etc.
    • Record the verdict.
  4. Write the report + verdict YAML.

Atom usage map (read-side only)

Schema sectionAtoms
per_programconnect_list_programs, connect_get_program
per_domain HQcommcare_list_apps, commcare_list_connections, commcare_list_inbound_apis, commcare_get_lookup_table, commcare_list_users (for user fields presence)
per_domain OCSocs_get_me, ocs_list_chatbots, ocs_inspect_chatbot
per_cohortcommcare_list_apps, commcare_get_lookup_table_rows, connect_get_opportunity, connect_list_payment_units, ocs_inspect_chatbot
per_usercommcare_list_users, commcare_get_user

Unverifiable rules (gap-tagged)

Per checklist-schema.yaml § atom_gaps, these rules currently grade unverifiable:

  • ucr-* rules — commcare_list_ucr_expressions atom ships, but verifier rule logic not yet wired (linked-domain push verification deferred)
  • repeater-* rules — a commcare-list-form-repeaters atom (not yet built) is needed (only create exists)
  • custom-user-data rule (cohort_id field exists) — commcare_list_user_fields atom ships, but verifier rule logic not yet wired
  • conditional-alert-payment rule — a commcare-list-conditional-alerts atom (atom code present in commcare.ts but registration deferred — see comment at line 510 of mcp/connect-server.ts) is needed

Closed since V1: ocs-interview-nodes-per-cohort is now verifiable via ocs_inspect_chatbot (router params.keywords[] are read directly).

For V1 these grade ⚠️ unverifiable with action-item prompts to operator. V1.5 ships the read atoms to fill these.

Exit codes

  • 0 — all rules pass or out_of_band
  • 1 — at least one fail
  • 2 — no fails but some unverifiable (operator should review action items)

Change Log

DateChangeAuthor
2026-05-21Initial V1 — read-only verifier; ~80% of rules implementable with current atomsACE team
2026-06-19Migrated every OCS rule (~8 rules) to a single ocs_inspect_chatbot call via OCS PR #3536. Closes the 24-hr TimeoutTrigger gap that drove our companion issue #3458. Adds ocs_get_me precheck for team scope.ACE team
rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).Stars: 196kUpdated Claude CodeCodexUnverifiedInstall
doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks.Stars: 196kUpdated Claude CodeCodexVerifiedInstall