Skip to content

ericrisco/rsc-harness / skills/constitution

constitution

Verified

Use when setting or amending a project's non-negotiables — stack canon, quality bars, conventions, security/a11y floors — as numbered, testable rules later phases obey, and drafting the REVIEW.md the reviewer reads. ASDD's base stage; writes 02-DOCS/wiki/sdd/constitution.md and REVIEW.md at the project root. NOT a feature spec (that is `specify`), NOT the technical plan (that is `plan`), NOT the wiki itself (that is `harness`).

FileKindSize
SKILL.mdskill md14.8 KB
evals/README.mdmarkdown3.6 KB
evals/cases.yamlconfig5.9 KB
references/constitution-template.mdmarkdown4.2 KB

evals/README.md

View on GitHub
# Eval harness — `constitution` skill

Two things are checked: that the skill **triggers** on the right prompts (and
stays quiet on near-misses that belong to a sibling), and that it **measurably
improves** the constitution it produces. Cases live in `cases.yaml`. There is no
shell runner — `constitution` is a process skill judged on safety rails and
judgment, so grading is done by an **agent harness** (a Claude Code agent with
skills loaded) plus a human spot-check.

## What's in `cases.yaml`

- `should_trigger` — prompts that MUST load `constitution`.
- `should_not_trigger` — near-misses that must route elsewhere (`route_to` names
  a real sibling in this repo: `harness`, `init`, `fastapi`, `security-audit`,
  `design`).
- `capability` — a scenario with a `must_include` rubric to grade with vs without.

## Triggering eval

Goal: the skill fires when it should and never on a near-miss.

1. Configure an agent with the **full catalog of skill descriptions** available
   for routing (constitution + its rsc-sdd phase siblings once they exist, plus
   harness, init, fastapi, nextjs, go, postgresdb, flutter, design, security-audit,
   deployment, marketing, presentations, course-storytelling, building-agents) so
   routing competes realistically.
2. For each `should_trigger` prompt: feed it cold, record whether `constitution`
   is the skill the agent loads. Run **3-5 trials** per prompt (fresh context).
3. For each `should_not_trigger` prompt: confirm `constitution` does NOT load and
   that the chosen skill matches `route_to`. Same 3-5 trials.
4. Score: `triggered_correctly / total_trials` across both lists.

**Pass bar: >= 90% trigger accuracy** over all prompts and trials, with **zero
systematic false-positives** on the `harness`/`init` near-misses — those are the
known traps, since "set up the project" reads close to "set the project's
principles". A scaffolding-or-bootstrap prompt must not pull in `constitution`.

## Capability eval

Goal: prove the skill changes the answer, not just the routing.

1. For the `capability` scenario, run it **twice**:
   - **WITHOUT** the skill (base agent, no `constitution` loaded).
   - **WITH** the `constitution` skill loaded.
2. Grade each output against the `must_include` checklist — one point per
   checkable item covered. A human or grading agent marks each present / absent.
3. Compute coverage = `items_covered / total_items` for each run.

**Pass bar: WITH the skill covers >= 80% of `must_include`; WITHOUT clearly
lower** (target a >= 30-point gap). The discriminating behaviors are: the
reconciliation pass against `02-DOCS/wiki/stack/*` (link, don't paste), testable
numbered principles, the fixed authorship + decisions-logged principles,
versioning + amendment log, the Definition-of-Done checklist, and explicit user
ratification. A base agent typically writes a vague, unversioned wall of
aspirations that duplicates the stack wiki — score that as a capability failure
even if it reads tidy.

## Notes on honesty

- Trials are stochastic; report the raw fraction, not a rounded "pass".
- The highest-signal capability check is reconciliation: a correct answer reads
  the existing stack articles and LINKS them; it never pastes their config into
  the constitution or silently resolves a contradiction between them.
- Re-run after any edit to `SKILL.md` or `references/constitution-template.md` —
  wording changes shift both triggering and rubric coverage.
- Several `route_to` targets are rsc-sdd phase siblings (`specify`, `plan`,
  `analyze`, `asdd`) that ship alongside this skill; the near-misses here route
  only to skills already present in the repo so the eval is runnable today.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall