hashgraph-online/awesome-codex-plugins / plugins/boshu2/agentops/skills/skill-eval
skill-eval
VerifiedMeasure whether a skill helps by comparing runs with and without it. Use when: reading skill A/B results or deciding to keep, revise or remove one.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.5 KB |
| references/behavioral-probes.md | markdown | 4.7 KB |
| references/coding-memory-readout.md | markdown | 2.9 KB |
| references/dialogue-skills.md | markdown | 1.2 KB |
| references/seeding.md | markdown | 5.7 KB |
references/behavioral-probes.md
View on GitHub# Behavioral probes: preserve their existing meaning
The AgentOps source checkout's `scripts/probe-skill.sh` remains the runner for
small behavioral regression
probes and immutable replay. It exposes an empty workspace and one injected
SKILL.md, not a complete installed-package coding trial. Its verdict measures
**behavior change**, never quality uplift or productive engineering completion.
Existing ledger entries retain that meaning and their recorded limitations.
| Probe form | Use when | Discriminator |
|---|---|---|
| Tier 1 — quiz | A decision rule is the caller's behavioral question | The answer/action on the scenario |
| Tier 2 — seeded task | Applying a discipline in work is the question | Whether the agent acted on a realistic planted defect |
Either form may be the starting point. Use [seeding](seeding.md) for seeded
tasks. Grade the act, never vocabulary copied from the treatment. A floor probe
detects at least one act; a multi-defect band needs both lower and upper bounds
to catch omission and finding spray. Calibrate against a transcript performing
the act without the prelude's wording and one repeating the wording without the
act.
The declared `treatment_source` remains the only arm variable: `canonical-skill`
uses exact SKILL.md bytes and is the mode the coverage gate counts;
`injected-prelude` establishes prelude-only evidence. The live runner uses the
Codex adapter; run it only when that adapter is available and caller-authorized,
with equal scenario and repetitions. An unavailable or unauthorized route is
UNMEASURED, not permission to substitute another runtime. Effort levels are a
declared experimental choice, not a prerequisite for every question.
Take the replay and live commands from the
[skill-probes README](https://github.com/boshu2/agentops/blob/main/evals/skill-probes/README.md#running)
(AgentOps source checkout only). Three
facts decide whether one runs and what it shows (checked 2026-10-25):
- **Replay needs a dated set.** Pass `--replay --fixtures <set>` naming a
directory that holds `fixture-set.json`; no probe has a usable default
`fixtures/` set, and `premortem-self-validation` (LEGACY-UNVERIFIED) has no
replayable set at all.
- **Exit 0 is not reproduction.** A replay reproduces a stored result only when
its scorecard reports `evaluator_matches_capture: true`. Every committed set
replays with `false`, and the coverage gate rejects each as ineligible by its
first failing check: the 2026-09-03 sets show `evidence-unverified` after the
warning "capture evaluator hashes do not match the exact repo-local evaluator
files"; `evaluator-stale` appears only once a set's recorded evaluator hashes
match. Treat the committed sets as historical.
- **Live reps and names are bound.** A live capture, only inside an authorized
envelope, takes `--reps` equal to `probe.json` `reps` (2 on every committed
probe) or omits it, plus a new `--fixtures` name and `--output` path each
time; the runner refuses another rep count, an existing fixture set or an
existing scorecard before dispatch.
AgentOps source checkout only: from the repository root,
`bash scripts/check-skill-probe-headroom.sh` reports the classes below.
Authoring a probe means supplying the package the runner reads (`probe.json`,
`question.md`, `discriminator.sh`, a prelude only in `injected-prelude` mode)
and its replay result.
## Headroom classifications
The existing `skill.probe-headroom` gate in `cli/internal/probeheadroom` owns
classification and thresholds. Its multi-effort saturation rule remains the
legacy gate contract; do not fabricate enough runs to satisfy it or rederive
the rule in a new report. Read and report the actual answer:
- **SATURATED:** the probe cannot distinguish the targeted act. Preserve the
observation as a scenario limitation in the RUNBOOK; do not append a skill
verdict to the legacy ledger. Do not infer skill value or lack of value.
- **FLOOR:** treatment did not act. Check the discriminator on a known passing
transcript. The result alone does not prove the skill cannot help elsewhere.
- **UNMEASURED:** no usable measurement, not INERT.
- **SEPARATED:** the gate found usable headroom. This classification itself does
not establish positive treatment benefit; retain the actual probe verdict.
## Legacy ledger
[`LEDGER.md`](https://github.com/boshu2/agentops/blob/main/evals/skill-probes/LEDGER.md) and
[`RUNBOOK.md`](https://github.com/boshu2/agentops/blob/main/evals/skill-probes/RUNBOOK.md)
(AgentOps source checkout only) keep the historical
verdicts under their own admissibility rules; keep a valid INERT or losing
result. A rep scored `infra` (discriminator exit 2) leaves the usable
behavioral rate but stays in the all-attempt count; zero usable treatment reps
is UNMEASURED, never INERT.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |