Skip to content

ericrisco/rsc-harness / skills/clarify

clarify

Verified

Use when a spec exists and must be de-risked before planning — run the gap checklist over its four ASDD questions and out-of-scope, ask only what changes what gets built (one question at a time, recommendation first), record every other call as an assumption, bake the answers back into the spec. The second half of ASDD stage 2 (Spec), between `specify` (writes the spec) and `plan` (designs the build). NOT the cross-artifact consistency check (that is `analyze`).

FileKindSize
SKILL.mdskill md14.0 KB
evals/README.mdmarkdown3.7 KB
evals/cases.yamlconfig6.0 KB

evals/README.md

View on GitHub
# Eval harness — `clarify` skill

These evals check two things: that the skill **triggers** on the right prompts
(and stays quiet on near-misses that belong to a neighbouring SDD phase), and
that it **measurably improves** the de-risking pass over a spec. Cases live in
`cases.yaml`. There is no pure shell runner — grading is a judgment call done by
an **agent harness** (a Claude Code agent with the skill catalog loaded) plus a
human spot-check.

## What's in `cases.yaml`

- `should_trigger` — prompts that MUST load `clarify` (a spec exists; the user
  wants it de-risked / hole-poked / readiness-checked before planning).
- `should_not_trigger` — near-misses routed to the genuinely correct sibling
  via `route_to` (`specify`, `plan`, `analyze`, `debug`, `constitution`).
- `capability` — a scenario with a `must_include` rubric to grade WITH vs
  WITHOUT the skill loaded.

## Triggering eval

Goal: the skill fires when a spec needs de-risking, and never on an adjacent
SDD phase.

1. Configure an agent with the **full catalog of skill descriptions** available
   for routing — the SDD chain siblings (`specify`, `plan`, `analyze`, `debug`,
   `constitution`, and the rest) plus the stack/process skills (`harness`,
   `init`, `fastapi`, `nextjs`, `go`, `postgresdb`, `flutter`, `design`,
   `marketing`, …) so routing competes realistically.
2. For each `should_trigger` prompt: feed it cold and record whether `clarify`
   is the skill loaded. Run **3–5 trials** per prompt with fresh context.
3. For each `should_not_trigger` prompt: confirm `clarify` does NOT load and
   that the chosen skill matches `route_to`. Same 3–5 trials.
4. Score: `triggered_correctly / total_trials` across both lists.

**Pass bar: ≥ 90% trigger accuracy** over all prompts and trials, with **zero
systematic false-positives** on the `specify` and `plan` near-misses — those are
the known traps. The boundary clarify must hold: no-spec-yet is `specify`,
how-to-build-it is `plan`. If it grabs either, the description is leaking.

## Capability eval

Goal: prove the skill changes the de-risking pass, not just the routing.

1. For the `capability` scenario, run it **twice**:
   - **WITHOUT** the skill (base agent, no `clarify` loaded).
   - **WITH** the `clarify` skill loaded.
2. Grade each output against the `must_include` checklist — one point per
   checkable item covered. A human or grading agent marks each present / absent.
3. Compute coverage = `items_covered / total_items` per run.

**Pass bar: WITH the skill covers ≥ 80% of `must_include`; WITHOUT clearly
lower** (target a ≥ 30-point gap). The discriminating behaviors are the ones a
base agent reliably misses: reading the constitution and citing what's already
resolved, ranking gaps by leverage instead of dumping every question, framing
questions as decisions-with-a-recommendation, and — the highest-signal item —
actually **baking answers back into the spec with a dated Clarifications log**
rather than just listing questions in chat.

## Notes on honesty

- Trials are stochastic; report the raw fraction, not a rounded "pass".
- The single highest-signal capability check is **the edit-back**: an answer
  that asks good questions but never modifies the spec or records the reasoning
  is a capability failure even if the questions are sharp. Clarify's deliverable
  is the sharpened spec, not the conversation.
- A second tell: a base agent often slides into proposing *how to build* the
  feature. A correct clarify pass stays on WHAT and defers HOW to `plan`. Treat
  architecture suggestions as a scope-leak failure.
- Re-run after any edit to `SKILL.md` — wording changes shift both triggering
  (especially the `specify`/`plan` boundary) and rubric coverage.
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall