Skip to content

ericrisco/rsc-harness / skills/harness

harness

Flagged

Use when governing a workspace's control plane, code or not — the `01-TOOLS/` tooling layer, the `02-DOCS/` chaos→knowledge wiki, the root Knowledge map. Audits it, migrates legacy `XX-*` folders, scaffolds provider tooling, sweeps the inbox, eats a file the person hands over ("cómete este fichero", "guárdalo en la documentación": moved into 02-DOCS/raw/ and documented), writes root CLAUDE.md/AGENTS.md. NOT the bootstrap front door (that is `init`, which hands off here).

evals/README.md

View on GitHub
# Eval harness — `harness` skill

This evaluates the `harness` skill on two axes: **triggering** (does it fire on
the right prompts and stay quiet on near-misses) and **capability** (does loading
it measurably improve the agent's output). These run via an **agent harness**
(an agent loop with skills loadable on/off), not a pure shell script — grading the
triggering and rubric coverage requires a model in the loop.

`cases.yaml` is the source of truth: `should_trigger` (7), `should_not_trigger`
(6), `capability` (2 scenarios).

## Triggering

1. Load **only** the `harness` skill into the agent (plus its declared siblings'
   *names/descriptions* so routing is realistic — do not load their bodies).
2. For each `should_trigger` prompt: feed it cold to the agent, run **3–5 trials**,
   record whether the agent invokes `harness`. Pass = it fires on every trial.
3. For each `should_not_trigger` prompt: same protocol. Pass = `harness` does
   **not** fire; bonus-check that it routes toward the `route_to` sibling (or
   correctly declines when `route_to: none`).
4. Score = (correct trigger decisions) / (total prompts × trials).

**Pass bar: >= 90% trigger accuracy** across all prompts and trials, with **no
single `should_not_trigger` prompt firing harness in a majority of its trials**
(a consistent false-positive is a hard fail even if the average clears 90%).

## Capability

For each `capability` scenario, run it twice with an identical agent:

- **WITHOUT** the skill (baseline) — skill not loaded.
- **WITH** the skill loaded.

Grade each transcript against that scenario's `must_include` checklist: one point
per checkable point that appears, correct and unprompted, in the output. A second
agent (or a human) grades; keep the grading rubric = the literal `must_include`
list so it stays reproducible.

**Pass bar:**
- WITH the skill covers **>= 80%** of the `must_include` points.
- WITH must **beat** WITHOUT by a clear margin (>= 30 percentage points). If the
  baseline already covers most points, the skill isn't adding value — investigate.

## Honesty notes

- These are **model-graded** evals; expect run-to-run variance. Always run
  multiple trials and report the spread, not a single number.
- Keep prompts **cold** — no system hint that the harness skill exists, or the
  trigger test is meaningless.
- Do **not** let the capability runs actually write to disk; the skill is
  destructive-by-consent. Run in a throwaway/sandbox workspace, or grade on the
  *proposed plan* (AUDIT output) rather than executed APPLY.
- When a case starts failing after a SKILL.md edit, fix the skill or the case —
  don't tune the grader to pass.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall