Skip to content

umararshad4/honey-stack / skills/write-skill

write-skill

Verified

Write a new honey-stack skill or patch an existing one to the quality bar, with evals.

FileKindSize
SKILL.mdskill md7.8 KB
agents/openai.yamlconfig154 B
references/evals.mdmarkdown4.0 KB
references/writing-for-agents.mdmarkdown1.4 KB

references/evals.md

View on GitHub
# Skill evals (`claude plugin eval` format)

Cases live in the repo's top-level `evals/` folder, one folder per case. The runner finds every `evals/<case>/prompt.md` with its `graders/*.md`.

```
evals/
  <situation>-uses-<skill>/      # should fire
    prompt.md
    graders/fired.md
    graders/<behavior>.md        # optional: what the skill must make the agent do
  <near-miss>-not-<skill>/       # should not fire
    prompt.md
    graders/not-fired.md
```

## prompt.md

YAML frontmatter, then the user's message exactly as a user would type it.

```markdown
---
max_turns: 4
tags: [skill-debug]
allowed_tools: [Read, Glob, Grep, Skill]
---

Our checkout test fails intermittently with `KeyError: 'currency'` about one run in five. Before changing anything, how should we track this down?
```

- `max_turns`: 3–4 for a trigger check; up to 8 when a behavior grader needs the agent to act.
- `tags`: free labels, e.g. `[skill-<name>]` or `[regression]`.
- `allowed_tools`: always include `Skill`, or the skill can't fire. Add `Bash`, `Write`, `Edit` only when a grader needs them.
- Write the prompt in the user's words and don't name the skill. The case tests whether the description's triggers catch real phrasing. Make it substantive: the agent skips skills on one-step asks it can answer directly, so a trivial prompt fails for the wrong reason.

## Graders

One file per grader, frontmatter only unless it's an `llm` grader.

**Should fire** (`graders/fired.md`). Replace `NAME` with the skill name; the optional prefix matches `honey:NAME` as well as `NAME`.

```markdown
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?NAME"'
---
```

**Should not fire** (`graders/not-fired.md`): the same match, with zero calls allowed on both arms.

```markdown
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?NAME"'
min: 0
max: 0
arm: both
---
```

**Behavior graders**, when the skill must change what the agent does:

| Type | Frontmatter | Body |
|---|---|---|
| `llm` | `type: llm` | One `PASS if …` line and one `FAIL if …` line, both concrete and observable |
| `regex` | `type: regex`, `pattern: "Checks run:"` | none; matches the last message |
| `tool_used` | `type: tool_used`, `tool: Bash`, `input_match: 'pytest'` | none; add `min`/`max` for counts |
| `file_exists` | `type: file_exists`, `path: "greet.py"` | none |

## Picking the cases

- **Should fire**: the most common real phrasing of the skill's main branch. For a patch, use the exact situation from the failure evidence.
- **Should not fire**: a near-miss. It shares keywords or a domain with the skill but needs something else; usually it belongs to a neighboring honey skill (a UI idea that isn't a bug for `debug`, a quick question that isn't a plan for `plan`). An obviously unrelated prompt tests nothing.
- **Regression** (patches): rebuild the failing session as a prompt, and grade the behavior that was missing. It must fail on the old skill.
- User-invoked skills can't fire on their own, so they get no trigger cases. Test their behavior by giving a fresh subagent the SKILL.md path and a task that tempts the failure, with a no-skill control run alongside.

## Running

From the repo root:

```bash
claude plugin eval . --case "<glob>" --runs 1 --ablation none --max-cost-usd 1 --no-publish
```

- `--case` takes a glob over case folder names, e.g. `"*-debug"` or `"bug-uses-debug"`.
- `--ablation none` skips the no-plugin arm. Under the default `with-without`, `tool_used: Skill` graders become with-only indicators and don't score.
- `--runs 1` is the cheap first pass. Before accepting a patch or calling a flaky result, re-run with `--runs 3`; one sample can mislead.
- Results land in `evals/results/<timestamp>/aggregate-result.json`: read `aggregates.casesPassed` and each case's `aggregates.passRate` and grader `explanation`. Add `--json <path>` for the full trace paths.
- For old-vs-new on a patch, run the same command on a checkout of each version and compare per case.
- On Codex, or anywhere without the `claude` CLI, record the eval as skipped with that reason; don't report it as passed.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 468kUpdated Claude CodeCodexUnverifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 319kUpdated Claude CodeCodexVerifiedInstall