tsunami28/MindNova / .claude/skills/governance-validate-skills
governance-validate-skills
VerifiedValidate the AI-SDLC skills under .claude/skills/ with the layered eval harness before relying on a new or changed one. Use when a skill has just been added or edited and needs checking, or when the user says "eval the skills", "validate this skill", "run the skill harness", "test the skills", "did my skill change break routing", or "check skill discriminability". This is the durable form of the "draft-only eval harness" the roadmap refers to. Do NOT use to validate the artifact graph or docs wiki (that is governance-check-graph), or to author or edit a skill's content.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.1 KB |
SKILL.md
View on GitHub---
name: governance-validate-skills
description: >
Validate the AI-SDLC skills under .claude/skills/ with the layered eval harness before relying
on a new or changed one. Use when a skill has just been added or edited and needs checking, or
when the user says "eval the skills", "validate this skill", "run the skill harness", "test the
skills", "did my skill change break routing", or "check skill discriminability". This is the
durable form of the "draft-only eval harness" the roadmap refers to. Do NOT use to validate the
artifact graph or docs wiki (that is governance-check-graph), or to author or edit a skill's
content.
---
# Validate Skills
Usable by **any** persona, typically Lead or Architect. Invoked as `/governance-validate-skills`.
A skill change is not done until it is validated. This is the sibling of `governance-check-graph`:
that gate validates artifacts and docs; this one validates the **skills themselves** - their
structure, how they route, and whether they add value over a competent baseline. It is
description-level and behavioural validation, not adoption.
Run it for any skill you add or change, and re-run the routing layer for the whole set when you
touch a description (routing is a shared space: one skill's triggers can collide with another's).
## The layers
| Layer | Question | Nature | How |
|---|---|---|---|
| 0 - static | Is every skill structurally sound and house-style clean? | deterministic | `MindNova/toolsCheck-SkillHygiene.ps1` |
| 1 - routing | Does each skill win its niche and avoid over-triggering? | subagent panel | independent routing judges over a prompt set |
| 2 - value | Does the skill beat a no-skill baseline on a real task? | subagent + blind judge | with-skill vs baseline, scored blind |
| 4 - cross-tool | Does the same SKILL.md trigger in Copilot CLI? | manual, off-Claude | cannot be driven from a Claude session; note as pending |
There is no Layer 3 here; the number is kept aligned with the roadmap's L0-L4 maturity language.
## Layer 0 - static (run first)
```powershell
MindNova/toolsCheck-SkillHygiene.ps1
```
Exits non-zero and names the offending skill and rule on any violation (name=folder, trigger-rich
description, persona line, Next-steps section, no em-dash, no line-citations, cross-refs resolve).
Fix and re-run until green before spending subagent budget on Layers 1-2.
To run this gate together with every other CA gate the pipeline runs (docs, code, story-traits,
plus the gate self-tests), use the local aggregator: `MindNova/toolsInvoke-LocalGates.ps1`
(add `-IncludeToolTests` to also run the Pester tests). It prints one pass/fail summary and exits
non-zero if any blocking gate fails, so it works as a pre-push check.
## Layer 1 - routing discriminability
Confirms each skill is the best pick for its own prompts and is NOT chosen for adjacent ones.
1. **Build the routing corpus** (what a routing agent actually sees - names + descriptions):
```powershell
Get-ChildItem .claude/skills -Directory | Sort-Object Name | ForEach-Object {
$d = Get-Content (Join-Path $_.FullName 'SKILL.md') -Raw
if ($d -match '(?ms)^description:\s*>?\s*(.+?)\r?\n---') { "### $($_.Name)`n$($matches[1].Trim())`n" }
}
```
2. **Write a prompt set** spanning three styles: straightforward (the skill's own trigger phrasing),
paraphrased (same intent, different words), and adversarial keyword-bait (a prompt that contains
the skill's keywords but whose true intent is a sibling skill - e.g. "we shipped X, create a
story" baits enrich-ticket but must route to elaboration-upsert-user-story). Cover the changed
skill's niche prompts AND its boundaries with each neighbour.
3. **Dispatch independent routing judges** (3 is a good default) as subagents. Each gets the corpus
and the prompts and picks the single best skill (or "none") per prompt from descriptions only.
4. **Score:** the skill should win every one of its niche prompts and be rejected on every bait
prompt; a tie/loss on a bait prompt is an over-trigger to fix (tighten the description or add an
explicit "Do NOT use" exclusion), then re-run.
## Layer 2 - value vs baseline
Confirms the skill earns its place against a capable agent that does not have it.
1. **Run two agents on the same representative task** (use a DRY RUN - report the plan, no live
writes): one **with-skill** (reads the SKILL.md), one **baseline** (no skill, but given the repo
conventions a real agent would have).
2. **Judge blind:** a third subagent scores both plans against a rubric without being told which is
which. Default rubric items (score 0-2 each): does it follow the skill's core discipline; does it
avoid the failure the skill prevents; is the output complete and correct; does it respect the
relevant gate or confirmation. Adapt the rubric to the skill under test.
3. **Read the result honestly:** skills win when they enforce a discipline the baseline drops; they
tie when that discipline has moved into a gate. A tie is a signal the durable value now lives in
the gate, not the prompt.
## Caveats (state them with results)
n is small (1 to 3 tasks, often 1 judge) - signal, not proof. Eval subagents may not inherit the
always-on `AGENTS.md` context a real session has, so baselines are slightly understated on always-on
rules. Layer 4 (Copilot cross-tool) is not runnable from a Claude session.
## Guardrails
- **Dry-run only.** Eval scenarios never perform live writes (no work item mutations, no PR, no file mutation in the
product). The agents report what they would do; a live write needs its own confirmation gate.
- **Record the outcome** in the Validation status section of `docs/ai-sdlc/roadmap.md`, dated, with
the per-layer result and the caveats. The roadmap is the durable record; chat threads are not.
- Pairs with `governance-check-graph` (artifact/doc integrity) and `governance-constitution`
(the non-negotiables a skill must respect).
## Next steps
- `/governance-check-graph` (Lead) - validate artifact and doc-link integrity alongside skill hygiene.
- If a routing or value finding requires changing a skill, edit it, then re-run this harness before relying on it.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |