ericrisco/rsc-harness / skills/specify
specify
VerifiedUse when a feature, change, or product idea is still fuzzy and must become an approved WHAT/WHY spec before any planning or code — the brainstorming front door of ASDD stage 2 (Spec). Turns a one-line intent into problem, goals, users, out-of-scope, the four ASDD questions (what the user can do, what result they get, what it must respect, how it proves it) and acceptance criteria, via frontier-round dialogue and 2-3 approaches with a recommendation, carrying zero implementation detail. Fires whenever someone jumps to HOW before WHAT is agreed, in any language. NOT the technical plan (that is `plan`), NOT the de-risking ambiguity sweep (that is `clarify`), NOT project-wide principles (that is `constitution`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 32.2 KB |
| evals/README.md | markdown | 3.8 KB |
| evals/cases.yaml | config | 10.4 KB |
| references/eliciting-requirements.md | markdown | 5.2 KB |
| references/spec-template.md | markdown | 5.8 KB |
evals/README.md
View on GitHub# Eval harness — `specify` skill
These evals check two things: that the skill **triggers** on the right prompts
(and stays quiet on near-misses that belong to other SDD phases or stack
skills), and that, when loaded, it **measurably changes the output** — a real
WHAT/WHY spec with no implementation leak, one-at-a-time questioning, the
canonical file path, and honest open points. Cases live in `cases.yaml`.
There is no shell runner. `specify` is a process skill judged on safety rails
and judgement, so grading is done by an **agent harness** (a Claude Code agent
with the rsc skill catalog loaded) plus a human spot-check.
## What's in `cases.yaml`
- `should_trigger` — prompts that MUST load `specify` (incl. Spanish and
non-obvious phrasings where the word "spec" never appears).
- `should_not_trigger` — near-misses that must route to the correct sibling via
`route_to` (`plan`, `clarify`, `constitution`, `tasks`, `fastapi`, `design`).
- `capability` — scenarios with a `must_include` rubric to grade with vs without.
## Triggering eval
Goal: the skill fires when an intent needs a spec, and never on the HOW, the
ambiguity sweep, or project-wide principles.
1. Configure an agent with the **full rsc skill catalog** available for routing
(the SDD phases — constitution, specify, clarify, plan, tasks, analyze,
implement, verify, review, ship — plus the stack/process skills: fastapi,
nextjs, go, postgresdb, flutter, design, marketing, presentations,
course-storytelling, building-agents, security-audit, deployment, harness,
init) so routing competes realistically.
2. For each `should_trigger` prompt: feed it cold, record whether `specify` is
the skill the agent loads. Run **3-5 trials** per prompt (fresh context).
3. For each `should_not_trigger` prompt: confirm `specify` does NOT load and the
chosen skill matches `route_to`. Same 3-5 trials.
4. Score: `triggered_correctly / total_trials` across both lists.
**Pass bar: >= 90% trigger accuracy** over all prompts and trials, with **zero
systematic false-positives** on the `plan` and `clarify` near-misses — those are
the known traps. A prompt about an *existing* spec (architecture, or "find the
ambiguities") must NOT pull in `specify`.
## Capability eval
Goal: prove the skill changes the answer, not just the routing.
1. For each `capability` scenario, run it **twice**:
- **WITHOUT** the skill (base agent, no `specify` loaded).
- **WITH** the `specify` skill loaded.
2. Grade each output against that scenario's `must_include` checklist — one
point per item covered. A human or grading agent marks each present / absent.
3. Compute coverage = `items_covered / total_items` for each run.
**Pass bar: WITH the skill covers >= 80% of `must_include`; WITHOUT clearly
lower** (target a >= 30-point gap). The skill must demonstrably add: the
no-implementation-leak discipline, one-question-at-a-time elicitation, the
canonical `02-DOCS/wiki/sdd/specs/<slug>.md` path + Knowledge-map index,
Given/When/Then acceptance criteria, and an honest *Points to clarify* handoff
to the `clarify` phase. If the base answer already scores ~80%, the case isn't
discriminating — tighten the rubric.
## Notes on honesty
- Trials are stochastic; report the raw fraction, not a rounded "pass".
- The highest-signal capability check is the **no-implementation-leak** rule: a
correct answer describes observable behaviour and refuses to name a framework,
table, or endpoint — pushing those to *Points to clarify*. Treat a confident
spec that already designs the solution as a **capability failure** even if it
reads well.
- The second-highest signal is **not over-asking**: a wall of questions, or
inventing answers to look finished, both fail the elicitation discipline.
- Re-run after any edit to `SKILL.md` or its `references/` — wording changes
shift both triggering and rubric coverage.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| planning-and-task-breakdownaddyosmani/agent-skillsBreaks work into ordered tasks. Use when you have a spec or clear requirements and need to break work into implementable tasks. Use when a task feels too large to start, when you need to estimate scope, or when parallel work is possible. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| screenpipethedotmack/claude-memSearch screen and audio history recorded by Screenpipe and optionally save selected context into claude-mem. Use when the user asks to recall desktop activity, a meeting, or something seen in another app through Screenpipe. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| contact-widgetnexu-io/open-designSelf-contained floating chat widget with welcome screen, social links, meeting button, and message input. Single HTML file, zero dependencies. | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |
| meeting-notesnexu-io/open-designMeeting notes page — title bar with attendees, agenda checklist, decisions block, action items table with owners + dates, and a "next meeting" footer. Use when the brief mentions "meeting notes", "minutes", "1:1 notes", "all-hands recap", or "会议纪要". | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |