Skip to content

ericrisco/rsc-harness / skills/python

python

Verified

Use when the task is Python itself, in any framework or none: PEP 695 generics, mypy --strict typing, dataclass/Protocol/TypedDict/Enum choices, asyncio.TaskGroup, stdlib idioms, src/ layout + pyproject.toml with uv, ruff+mypy+pytest gate. NOT a FastAPI/ASGI service (that is fastapi), NOT a deep pytest suite (that is testing-py).

FileKindSize
SKILL.mdskill md14.9 KB
evals/README.mdmarkdown1.4 KB
evals/cases.yamlconfig3.5 KB
references/async.mdmarkdown5.3 KB
references/stdlib.mdmarkdown5.1 KB
references/typing.mdmarkdown5.8 KB
scripts/verify.shexecutablescript4.1 KB

evals/README.md

View on GitHub
# Eval harness — `python` skill

These cases are read by an **agent harness** (a Claude agent with the skill catalog loaded),
not a plain shell script. `cases.yaml` is the fixture; this file is the procedure. Two things
are measured: **triggering** (does `python` fire on language-level prompts and stay silent on
near-misses that belong to `fastapi`, `testing-py`, `django`, `security-audit`, `deployment`?)
and **capability** (does loading the skill make a scaffold answer measurably better?).

**Triggering:** load the full sibling descriptions, start a fresh session per prompt, feed each
`should_trigger.prompt` and confirm `python` fires; feed each `should_not_trigger.prompt` and
confirm it routes to the stated `route_to` and does *not* fire `python`. LLM routing is
non-deterministic — run 3–5 trials each and pass on the majority; target ≥90% accuracy.

**Capability:** run the `capability.scenario` twice — once with the skill body unavailable, once
with `python/SKILL.md` (and references) loaded — and grade each output against the `must_include`
rubric (fraction of points present and correct, averaged over 3 trials). Pass bar: ≥80% with the
skill, and a clear margin over the without-skill baseline. No network is required; grading is
rubric-based and partly judgment (e.g. "no mutable default argument"). Keep `cases.yaml` faithful
to SKILL.md's "When to use / When NOT to use" — change them together.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall