hashgraph-online/awesome-codex-plugins / plugins/boshu2/agentops/skills/skill-eval
skill-eval
VerifiedMeasure whether a skill helps by comparing runs with and without it. Use when: reading skill A/B results or deciding to keep, revise or remove one.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.5 KB |
| references/behavioral-probes.md | markdown | 4.7 KB |
| references/coding-memory-readout.md | markdown | 2.9 KB |
| references/dialogue-skills.md | markdown | 1.2 KB |
| references/seeding.md | markdown | 5.7 KB |
references/coding-memory-readout.md
View on GitHub# Coding and memory readout
Readout rules for controlled coding comparisons and memory transfer tests. Use
the development adapter documented in
[`evals/skills-rpi/readout.md`](https://github.com/boshu2/agentops/blob/main/evals/skills-rpi/readout.md)
(AgentOps source checkout only), or the
caller's existing equivalent. Its report is a rebuildable view, not work
authority. The pilot's default `insufficient-evidence` recommendation is an
honest limit; the specialist may make a narrower supported maintenance
recommendation and must state its evidence and provisional scope.
- Report endpoint success against **all assigned/observed attempts** alongside
any feasible-task rate. Retain infrastructure invalidity, infeasibility and
unknown coverage separately; do not hide them by dropping the denominator.
- Report false completion, false acceptance and needless blocking separately
when independent evidence measures them. Clean cases and abstentions are
denominators, not opportunities to reward finding-count spray. Unknown is not
zero. Deterministic code truth may settle an experimental criterion, while a
required native handoff or exact-subject judgment remains unproven.
- Report raw time/cost distributions and total cost of all attempts per accepted
outcome. Zero accepted outcomes makes that ratio undefined. Keep counters
from different sources distinct and never sum them (the pilot readout owns
its Harbor and native counter rules); split cost by phase only where native
identity supports it. State the measurement window and excluded setup and
analysis overhead; use actual input accounting, not context freshness.
- `evals/_stats` (needs numpy and scipy) computes a paired task-cluster
bootstrap for paired scores; it is not the Wilson/Fisher readout for two
unpaired rates, and its docstring still describes a removed CLI caller.
A pilot is descriptive unless sample size and
decision thresholds were justified and fixed in advance. A zero-crossing
interval or `no_change` is **not equivalence**; equivalence needs its own margin
and test. Same numeric repetitions/seeds do not prove controlled provider
randomness. Do not extrapolate local results across libraries or models.
- For memory, hold skill/runtime fixed and compare frozen with independently
qualified updated memory in fresh later sessions, using an unseen transfer
task and an unrelated or invalidating control. Count acquisition, qualification,
retrieval and downstream trial cost separately. Package available, content
delivered, relevant action and later outcome are separate facts. Saving a page
earns no benefit credit; coding-pilot completion does not establish compounding.
Storage follows [SKILL.md](../SKILL.md#procedure); preserve legacy `.agents/`
evidence. Independent support and disclosure review precede any memory import;
do not auto-publish transcripts or change knowledge from aggregate scores
(ADR-0016).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |