hashgraph-online/awesome-codex-plugins / plugins/boshu2/agentops/skills/skill-eval
skill-eval
VerifiedMeasure whether a skill helps by comparing runs with and without it. Use when: reading skill A/B results or deciding to keep, revise or remove one.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.5 KB |
| references/behavioral-probes.md | markdown | 4.7 KB |
| references/coding-memory-readout.md | markdown | 2.9 KB |
| references/dialogue-skills.md | markdown | 1.2 KB |
| references/seeding.md | markdown | 5.7 KB |
references/seeding.md
View on GitHub# Seeding a forcing defect (tier 2)
A **forcing defect** is a flaw planted in a realistic work artifact that the
skill's discipline catches and a skim does not. The probe grades whether the
agent *acted* on it.
## Diagnostic calibration
A seeded probe can help diagnose headroom. These are possible observations,
not requirements for accepting or retaining a case:
```
too obvious USABLE WINDOW too obscure
| | |
both arms control misses, neither arm
catch it <------- treatment catches ----> catches it
(ceiling) (floor)
```
Ceiling and floor observations limit the question a probe can answer. Calibrate
the discriminator on known transcripts before spending live reps. The check:
**can the defect be derived from the discipline
alone?** If catching it needs domain trivia the skill never taught, it is below
the window. If catching it needs nothing but reading the first paragraph, it is
above.
## The four seed shapes
Ordered by how reliably they escape saturation.
### 1. Buried in a green context
The defect sits inside output that otherwise reads as success. A test summary
where 47 pass and one is quietly `skipped`. A scanner log whose middle line says
`0 rules loaded` above a triumphant `0 findings`. A migration report that lists
every table as `ok` except one marked `deferred`.
This is the strongest shape because it attacks the actual failure mode — an
agent that pattern-matches "looks green" and stops.
### 2. Euphemized
The defect is present and correctly described, but in language that does not
trip the obvious keyword. `not_checked` rendered as "covered by existing
behavior." A self-graded close written as "verified by the implementing lane."
An unbounded write scope described as "touching the relevant files."
Attacks keyword-matching rather than comprehension. Pairs well with a
discriminator that grades the act, since a keyword-matching agent will not
produce the act.
### 3. Structural, not local
No single line is wrong. The defect is the *shape*: every unit of work in a plan
is verified by the context that authored it. Nothing is false; the arrangement
is. Requires the discipline to see, which is exactly what tier 2 measures.
### 4. Under time pressure
The scenario states a deadline, a release window, or a waiting stakeholder. This
does not add a defect — it lowers the threshold at which the agent accepts the
green reading. Use as a **modifier** on shapes 1–3, never alone.
## Rules
1. **One defect per floor probe.** Two defects and a floor assertion cannot tell
"caught both" from "caught one and got lucky."
2. **N defects for a band probe, and N must be exact.** `probe.json`'s
`seeded_defects` must equal what is actually in `question.md`. A drifted count
makes every band assertion meaningless and nothing will catch it.
3. **Defects must be independent.** If catching defect A makes B obvious, the
band is really N−1 and the lower bound is wrong.
4. **The artifact must be work, not a quiz.** No "review this and tell us what's
wrong." Hand the agent the task the skill exists for; the defect is in the
material.
5. **Keep it short.** Under 40 lines. A long artifact measures reading stamina.
6. **Never signpost.** No "note the following" or "one of these is incorrect."
Signposting converts a tier-2 probe back into a tier-1 quiz.
7. **The defect is realistic.** It should be something a real contributor would
plausibly write. Implausible defects get caught by implausibility, not by the
discipline.
## Optional live development calibration
Use live calibration only inside an already accepted trial/time envelope. It
is development data, counts against the total cap, and does not become a hidden
holdout. Predeclare its stop; do not keep reseeding until treatment wins.
1. Draft the artifact with the defect.
2. If needed for the decision, run a bounded control sample at the deployed
model and selected effort. Two reps can diagnose a scenario, not prove a rate.
3. If the control catches it in every sampled rep, record the observed ceiling.
Keep it as an easy regression/cost control when relevant. A new development
variant is a separate version, not permission to erase an unfavorable case.
4. If the control arm catches it in **0/2**, hand the same artifact to the
treatment arm. If treatment is also 0/2 — below the window. The defect is not
showing a positive signal in this sample. Check the discriminator and record
the floor; do not infer a general defect in the skill from two misses.
5. Report the observed results whether positive, null or adverse. Freeze the
chosen development cases before any separately authorized comparison.
Count all calibration starts, including failures and treatment calibration;
calibration does not always cost only two reps or guarantee useful separation.
## Worked shape (illustrative)
For a skill whose discipline is *disclose unverified acceptance surface*:
- **Artifact:** a short closeout report for a merged change. Four acceptance
criteria, each with an evidence line. The third criterion's evidence reads
"behavior unchanged from prior release, so existing coverage applies."
- **Shape:** euphemized (2) — that sentence is `not_checked` wearing a
justification.
- **Action graded:** does the agent return `NOT_PROVEN` and name criterion three,
or does it return `PASS`?
- **Why it sits in the window:** the sentence is plausible and reads as diligence.
Catching it requires applying the rule *that a bounded proof is not a proof of
the criterion* — derivable from the discipline, invisible to a skim.
Do not copy this artifact into a probe. It is here to show the reasoning; a
scenario reused across probes trains toward itself.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 469k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |