Adonis0123/adonis-skills / skills/agent-skill-evaluation
agent-skill-evaluation
FlaggedUse when comparing or trial-testing agent skills before installing — including GitHub/skills.sh URLs, 测试一下/值不值得装, package screening, and A/B artifact runs.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 30.3 KB |
| references/conceptual-multi-agent-skill-trial.md | markdown | 4.0 KB |
| references/image-producing-skill-trial.md | markdown | 2.7 KB |
| references/mattpocock-catalog-renames.md | markdown | 2.7 KB |
| references/mattpocock-shortlist-20260807.md | markdown | 3.2 KB |
| references/mattpocock-skills-screening.md | markdown | 15.3 KB |
| references/mattpocock-tdd-trial-20260807.md | markdown | 2.8 KB |
| references/mattpocock-tdd-trial-20260908.md | markdown | 2.5 KB |
| references/mattpocock-to-prd-trial-20260807.md | markdown | 1.8 KB |
| references/mattpocock-wayfinder-trial-20260909.md | markdown | 2.3 KB |
| references/output-style-skill-screening.md | markdown | 6.0 KB |
| references/peer-review-cli.md | markdown | 5.4 KB |
| references/single-skill-temp-trial.md | markdown | 7.7 KB |
| references/skill-eval-harness-trial.md | markdown | 3.6 KB |
| references/tdd-process-skill-trial.md | markdown | 2.9 KB |
| references/ui-design-ab-testing.md | markdown | 5.8 KB |
references/skill-eval-harness-trial.md
View on GitHub# Skill eval-harness trial
Use when the candidate is a **CLI that grades Agent Skills** (declarative YAML cases, a real Agent engine, rule/script/LLM judges, reports) plus an optional companion skill that drives that CLI. This is not a skill-install decision and not a UI A/B.
## Classify first
| Shape | Route |
| -------------------------------------------------------------------------------------------- | ----------------------- |
| One `SKILL.md` the user might enable | Single-skill temp trial |
| Router + specialist skills | Package screen |
| Binary + `eval.yaml` / `cases/` + engine adapters, companion skill only orchestrates the CLI | **This file** |
The companion skill is not the product. Do not `npx skills add` it to try the harness. Keep the binary under `/tmp/<name>-eval/` and invoke it by absolute path.
## What job it actually does
It answers: **after this Skill is loaded into Claude Code / Codex / another engine, does the behavior still match the cases?**
It does **not** answer: should we install this Skill, is our install-screening skill worse, or does a keyword in the reply mean the Skill was followed.
Default lean **B** (do not enable the companion; do not put the binary on `PATH`) unless a recurring Skill already has cases and the user wants regression on every edit. Local install-screening stays the pre-install path.
## Smoke (do this, in order)
1. **Release binary into `/tmp` only.** Prefer the published checksummed archive for the host OS/arch over `go install` or `curl | bash` into `~/bin`. Confirm `command -v` does not resolve the new name, and that neither skill dir gained the companion.
2. **Write one tiny fixture Skill** under the trial tree: `SKILL.md` + `evals/eval.yaml` (`schema_version: v1alpha1`, `environment.type: none`, `engine.name` = a CLI already on the machine) + one `evals/cases/<id>.yaml`.
3. **`validate` then `run --dry-run`.** A valid suite prints the loaded case count. A deliberately broken `schema_version` must fail. Dry-run must list the case and must not start the engine.
4. **One live case, then stop.** Empty model + no API key is valid when the docs say the engine falls back to host login — the first real case is the login check, not `validate`. Do not loop cases to "get a better score."
5. **Read the graded response, not only PASS.** Open `result.json` `case_results[].response` (or the engine transcript). Record duration and token totals from that file.
## Pitfalls
- **A keyword assertion is not behavior proof.** `expect.must_contain` / `output_contains` passes if the token appears anywhere, including "I will not do that." Count the case only when the assertion would fail if the Skill were ignored.
- **`environment.type: none` is the host.** The engine CLI runs with its own tools and login. Inspect the transcript for reads or writes outside the trial workspace before claiming the run was contained. Isolated regression uses the harness's container/sandbox runtime, not `none`.
- **Do not treat the companion's "you do not need to install the CLI" as the trial path.** That sentence is for an agent that already has the companion loaded. A 测试 turn still proves the binary itself.
- **Token cost is part of the verdict.** One short case can spend a large input budget because the engine loads its own context. Report input/output tokens next to PASS/FAIL; do not recommend suite-wide runs from a dry-run alone.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |