dimagi-internal/ace / skills/detect-structure-drift
detect-structure-drift
UnverifiedAudit the ACE plugin's internal structure for drift between the three sources of truth — agent frontmatter, lib/artifact-manifest.ts, and agents/orchestrator-reference.md § State Schema. Reports unclaimed manifest producers, orphan SKILL.md directories, missing skill files, phase-vocab mismatches, and state-schema parity gaps.
detect-structure-drift
The ACE plugin's lifecycle is described in three machine-or-human-readable places that should agree but can drift:
- Agent frontmatter —
agents/<phase-agent>.mdYAML headers. Each phase agent declaresphase,phase_ordinal,phase_display, and orderedskills:+recurring_skills:+manual_skills:arrays with nestedqa_skill:,eval_skill:,has_judge:properties. This is the structural spine —apps/system/reader.pyin ace-web parses this for the /system page. (manual_skills:is the HITL track — e.g.solicitation-reviewinsolicitation-management.md. Treat its entries as phase-claimed for every check.) lib/artifact-manifest.ts— every artifact a run writes to Drive, withproducedBy: <skill-or-agent>+phase: <name>. The execution roster: the broadest enumeration of skills that actually do work in/ace:run.agents/orchestrator-reference.md§ State Schema — proserun_state.yamltemplate showing per-phase skill ordering. Human-curated documentation that the orchestrator writes against.
This skill compares them and surfaces drift. It is read-only and idempotent.
Inputs
$CLAUDE_PLUGIN_ROOT— the ACE plugin install root (Claude Code sets this when invoking plugin commands; fall back to the cwd if unset and the user invoked from a checkout).- Optional first arg
--strict— fail loudly (annotateEXIT STATUS: 1at the bottom of the report) if any check produces a FAIL. Default is report-only.
Process
Step 1: Resolve plugin root
Determine $PLUGIN_ROOT. If $CLAUDE_PLUGIN_ROOT is set, use it. Else walk up from $PWD looking for a directory containing both agents/ and skills/ and lib/artifact-manifest.ts. If neither resolves, abort with "Cannot find ACE plugin root; set CLAUDE_PLUGIN_ROOT or run from a checkout."
Step 2: Read the three sources
Issue these reads in parallel so all the content is in context for cross-checks:
- Every file matching
$PLUGIN_ROOT/agents/*.md— Read each. Parse the YAML frontmatter (between the first two---lines). Collect for every agent:name,phase,phase_ordinal,phase_display,skills(list of objects),recurring_skills(list of objects),manual_skills(list of objects). Skipagents/orchestrator-reference.md— it's the state-schema source for Check E, not an agent (it has an empty frontmatter block). Agents without aphase:field are non-phase agents — currentlyace-orchestrator,ocs-tester,sweep. Non-phase agents that DO have askills:array (e.g.ocs-testerre-claimingocs-chatbot-qa/ocs-chatbot-eval) still contribute their names to thephase_claimedset for Check C purposes — those skills are reachable via that agent. $PLUGIN_ROOT/lib/artifact-manifest.ts— Read. Extract:- The
export const PHASES = [...]array (the canonical phase vocab). - Every
producedBy: '<name>'occurrence with its surroundingphase: '<name>'value. Buildmanifest_producers: Map<producer_name, set<phase>>.
- The
$PLUGIN_ROOT/lib/artifact-manifest-roles.ts— Read. ExtractPHASE_FOLDERSkeys.$PLUGIN_ROOT/agents/orchestrator-reference.md— Read. Find the firstphases:YAML block (the State Schema section starting around line 44). For each<phase>:heading and its child<skill>: <status>lines, collectstate_schema: Map<phase_name, list[skill_name]>. Stop at the first non-indented line after the block.- Use Glob (or
lsvia Bash) on$PLUGIN_ROOT/skills/*/SKILL.mdto enumerate every SKILL.md directory. Record the setskill_dirs.
Step 3: Build the canonical phase spine
Sort agents that have a phase_ordinal by ordinal. Emit the phase spine table:
PHASE SPINE (agent frontmatter):
ordinal phase phase_display agent
------- ----------------------------- ----------------------------------- -----------------------------
1 idea-to-design Idea to Design idea-to-design
2 scenarios-and-acceptance Scenarios & Acceptance Planning scenarios-and-acceptance
...
10 closeout Closeout closeout
NON-PHASE AGENTS:
ace-orchestrator, orchestrator-reference, ocs-tester, sweep
Step 4: Run the drift checks
Check A — Skill claims have backing files. For every name referenced from agent frontmatter (the entry's name, plus qa_skill, plus eval_skill when present, across skills: + recurring_skills: + manual_skills:), assert $PLUGIN_ROOT/skills/<name>/SKILL.md exists. A missing file is a FAIL. Format:
A — Skill claims have backing files
PASS — all <N> claimed skills have SKILL.md
-- or --
FAIL — idea-to-pdd-qa: claimed by idea-to-design (as qa_skill of idea-to-pdd) but skills/idea-to-pdd-qa/SKILL.md does not exist
Check B — Manifest producers are claimed. For every distinct producedBy: value in the manifest:
- Skip the literal string
'external'(human-provided inputs). - INFO-mark any
producedBy:value that equals the name of a phase agent from the spine OR the name of a non-phase agent (ace-orchestrator,ocs-tester,sweep). These are bookkeeping artifacts owned by the agent itself, not by a skill — don't flag them as unclaimed. - For every remaining producer, assert it appears in some agent frontmatter — either as a top-level
skills[].name/recurring_skills[].name/manual_skills[].name, or as a nestedqa_skill:/eval_skill:value. Unclaimed = WARN. Format:
B — Manifest producers are claimed
WARN — decisions-render: produces artifacts in phase 'design' but no agent frontmatter claims it
INFO — ace-orchestrator: agent-owned bookkeeping artifact (run-readme, etc.)
Check C — SKILL.md directories are reachable. For every directory in skill_dirs, classify it. Order matters — apply the first rule that matches:
- Phase-claimed if it appears as
namein some agent'sskills/recurring_skills/manual_skills. - Judge-claimed if it appears as
qa_skilloreval_skillin some agent's frontmatter. - Manifest-only if it appears only as a
producedBy:inartifact-manifest.ts. - Command-claimed if some
commands/*.mdbody referencesSkill(<name>)or/ace:<name>— these are slash-command-invoked utility skills (e.g.sweep-*,fork-run,upload-transcript,interview-*,video-from-program-page). - Sweep-claimed if
agents/sweep.mdbody names the skill as a dispatch target (the sweep umbrella dispatches its system-specific children inline). - Orphan otherwise.
Print each non-phase-claimed group explicitly. Orphans are WARN; manifest-only, command-claimed, and sweep-claimed are INFO. The point of the three INFO buckets is to distinguish why a skill isn't in the agent frontmatter — it tells the operator whether the gap is benign:
C — SKILL.md directories are reachable
PASS — <N> total, <P> phase-claimed, <J> judge-claimed
INFO — manifest-only (run by phase but not in agent frontmatter): decisions-render, eval-calibration, ...
INFO — command-claimed (invoked via /ace:<X>): fork-run, upload-transcript, interview-*, ...
INFO — sweep-claimed (dispatched by agents/sweep.md): sweep-drive, sweep-connect, ...
WARN — orphan (no agent, no manifest, no command, no sweep ref): skills/<name>/, ...
Check D — Phase vocab consistency. Build three sets:
agent_phases= set ofphase:values across all phase agents.manifest_phases=PHASESarray fromlib/artifact-manifest.ts.folder_phases= keys ofPHASE_FOLDERSfromlib/artifact-manifest-roles.ts.
They should all be equal modulo a closed set of intentional aliases. The manifest uses short folder-prefix-stem names; the agent uses full phase ids. The allowlist is:
ALLOWLISTED_ALIASES = { # (manifest_phase, agent_phase) pairs — mirror lib/artifact-manifest-roles.ts § PHASE_FOLDERS
('design', 'idea-to-design'),
('scenarios-and-acceptance', 'scenarios-and-acceptance'),
('commcare', 'commcare-setup'),
('connect', 'connect-setup'),
('ocs', 'ocs-setup'),
('qa-and-training', 'qa-and-training'),
('synthetic-data-and-workflows', 'synthetic-data-and-workflows'),
('solicitation-management', 'solicitation-management'),
('execution-management', 'execution-management'),
('closeout', 'closeout'),
}
Treat the manifest and agent vocabularies as equal under this mapping. Flag any phase that doesn't pair under the allowlist as FAIL. If a new phase is added in the future, extend the allowlist here with a one-line comment explaining why — don't loosen the check.
D — Phase vocab consistency
PASS — all <N> phases pair under the allowlist
-- or --
FAIL — phase 'foo' present in artifact-manifest.ts but no agent-phase pairs with it
Check E — State-schema parity. For each <phase>: in state_schema from orchestrator-reference.md:
- Assert
<phase>appears inagent_phases. Missing = FAIL. - For each
<skill>:child line under it, strip any trailing-quick,-monitor, or-deepsuffix before lookup — these are step-disambiguation tags (Phase 5 vs Phase 9 vs/ace:qa-deepinvocations of the same underlying skill). Then assert the stripped name appears either as anameorqa_skilloreval_skillsomewhere in agent frontmatter, OR as aproducedBy:in the manifest. Unclaimed step = WARN (likely the prose drifted from the structural sources).
Conversely, for each agent's skills[].name, assert it appears (with or without the suffix) under the matching <phase>: block in the state schema. Skill missing from the state schema = WARN.
Whole-phase missing from state schema = FAIL, not WARN — that's a missing template stanza, not prose drift. Surface it as its own bullet at the top of Check E output, separate from per-step warnings.
E — State-schema parity (orchestrator-reference.md § State Schema)
PASS — all <N> documented steps map to known skills
WARN — orchestrator-reference lists 'foo' under 'qa-and-training' but no agent or manifest claims it
WARN — agent 'qa-and-training' claims skill 'bar' but it's missing from the state schema
Step 5: Render the report
Print to stdout in this order:
- Header line:
# detect-structure-drift @ <PLUGIN_ROOT> — VERSION <X.Y.Z>(readVERSIONfile). - Phase spine table (Step 3).
- Each check (A-E) as its own section in order. Within each section, print the result line first (PASS/WARN/FAIL summary), then any detail rows.
- Counts footer:
Findings: <FAIL> FAIL, <WARN> WARN, <INFO> INFO across <C> checks. - If
--strictwas passed AND any FAIL was reported: appendEXIT STATUS: 1. ElseEXIT STATUS: 0.
The report is the only side-effect. Do not modify any file. Do not propose fixes inline — the operator decides whether each drift is a bug or an intentional gap (e.g. a manifest-only utility skill might be deliberate cross-phase plumbing).
When to use
- Before a release — catches the case where a SKILL.md was deleted but its
qa_skill:reference in an agent wasn't (FAIL on Check A). - After adding a new skill — verifies the skill is claimed somewhere visible (Check C makes the choice explicit: phase, judge, manifest-only, or orphan).
- After renaming a phase — Check D catches alias drift across the three files.
- Investigating "why is skill X showing in Utility Skills on /system?" — Check C's manifest-only and orphan groups answer it directly.
Out of scope
- Cross-repo checks. This skill audits the ACE plugin only. Whether
ace-web'sapps/system/reader.pydisplays the structure correctly is a separate concern (the reader currently drops nestedqa_skill:/eval_skill:from the phase display, which is why some skills look "missing from phases" on/system). If you want to audit ace-web's rendering, write a sibling skill there. - Eval-grade contents of SKILL.md. This skill doesn't grade prompt quality, only structural references.
- Mechanical fixes. No auto-rewrites. The output is for a human (or a follow-up skill) to act on.
Implementation notes for agents
- The artifact manifest is ~1290 lines but uniform. A regex over the source —
/producedBy:\s*'([^']+)'/g— is sufficient; do not try to parse it as TypeScript. Pair each match with the nearestphase: '...'on the same{...}block by scanning backwards from the match position. - For the orchestrator-reference State Schema block, anchor on the first occurrence of
^phases:$and read until the next blank line followed by a non-indented heading or fence. There are multiplephases:blocks in the file (currently at lines 44, 501, 801) — the first isrun_state.yaml's schema; the others are inline examples fordecisions.yamletc. Always use the first. - When parsing agent frontmatter inline-flow objects (e.g.
- { name: foo, has_judge: true, qa_skill: foo-qa }), find the closing}via brace-depth counting first, then split the inside on top-level commas. Trailing#line comments after the closing}(e.g.} # canopy:walkthrough scores per scene) must be discarded — they appear after several entries insynthetic-data-and-workflows.mdandscenarios-and-acceptance.md. Values may have arbitrary horizontal whitespace before them (column-aligned style varies file-to-file). - The phase-name allowlist for Check D (the manifest⇄agent alias set) is the ONLY substitution table. If a new phase is added, extend
ALLOWLISTED_ALIASESabove with a one-line comment explaining why — don't loosen the check. - Run all Reads in parallel where possible; the audit should complete in under a few seconds even with 100+ skill dirs.
Related
apps/system/reader.py(inace-web) — the parser that produces the /system page. Currently does NOT promote nestedqa_skill:/eval_skill:entries into phase rows for display; they fall to "Utility Skills." This skill makes that gap visible via Check C's manifest-only and judge-claimed groupings.lib/artifact-manifest.ts§ header comment — names this skill's #2 source-of-truth explicitly: "single source of truth for ... which skill produces each artifact."agents/orchestrator-reference.md§ State Schema — the human-curatedrun_state.yamltemplate that Check E grades against.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |