Skip to content

rlaope/oh-my-hermes / skills/omh-agent-evaluation

omh-agent-evaluation

Verified

[omh] Choosing between coding agents on evidence: compare executor or agent choices on reproducible tasks using quality, cost, time, tool, and evidence metrics. Use when the user says: agent-evaluation, agent evaluation, agent eval, agent benchmark, executor evaluation, executor benchmark, compare agents, compare codex claude.

FileKindSize
SKILL.mdskill md9.7 KB
references/self-evaluation-loops.mdmarkdown6.2 KB

references/self-evaluation-loops.md

View on GitHub
# Self-Evaluation Loops

Comparing two executors is one question; an agent judging and improving its
own output is another. This is the contract for the second: which loop shape
to use, when it stops, and what a score from it is worth.

## The three shapes

| Shape | Who judges | Use it when |
| --- | --- | --- |
| Reflection | the same model, on its own output | no external check exists and the criteria are stylistic or structural |
| Evaluator-optimizer | a separate evaluator scores against written criteria, an optimizer revises | quality matters enough to pay for two passes and the criteria can be written down |
| Test-driven refinement | an executable check - tests, a type checker, a schema, a linter | the output is code or data whose correctness a machine can decide |

**An executable check outranks a judge whenever one exists.** A test that
fails is a fact; a judge that says "looks good" is an opinion generated by
the same class of process that produced the output. Reaching for a rubric on
code that could simply be run is the most common way these loops produce
confident garbage.

## Stop rules are a contract, not a default

Every loop carries all three, declared before it starts:

1. **A maximum iteration count.** Three to five; a loop with no ceiling is a
   budget leak, not a quality strategy.
2. **A score threshold** - the value at which the output is good enough. It
   is chosen before the first run, because a threshold set afterwards is a
   description of the score that happened.
3. **A no-improvement break.** If an iteration does not improve the score,
   stop: the loop has converged, and further passes usually rewrite rather
   than improve.

A loop whose only stop is "it looks good now" is a defect. Report the
iteration count, the final score, and **which of the three rules ended it** -
a run that hit the ceiling is a different result from one that cleared the
threshold, even when the output looks the same.

## Criteria and rubrics

Criteria are written **before generation**, not after reading the output;
criteria derived from an output describe it rather than test it. A rubric
names its dimensions and their weights up front, scores each dimension
separately, and reports the dimension scores next to the total - a single
number hides which dimension failed, which is the one thing the loop needed
to know.

## When a model may judge

- The criteria exist in writing and the judge scores against them, not
  against preference.
- **Self-judgement is the weakest evidence class in the ladder** - the same
  model, the same blind spots. It is a signal, and it is labelled as one.
- A judge score is never correctness. It does not license a claim that the
  output is right, tested, reviewed, or shippable, and the loop's own report
  says so rather than leaving the reader to assume it.
- Comparing two outputs is easier for a judge than scoring one absolutely;
  when a baseline exists, prefer the comparison and record which side was
  which, since presentation order alone moves judgements.
- Keep the full trajectory: every iteration's output, score, and critique.
  A refinement loop with no history cannot be debugged, only rerun.

## Qualifying a judge

A judge nobody measured is unqualified, and an unqualified judge's number is
not a result. Qualification happens before the run whose score depends on it,
and again whenever the grader model ID, the rubric, or the task distribution
changes - a judge qualified on last quarter's tasks is unqualified on these.

1. **Label a sample by hand.** Fifty cases is a working floor. Draw them from
   the distribution the judge will actually score, and include the boundary
   cases the rubric is likeliest to get wrong; a sample of clear passes and
   clear failures qualifies a judge for a job it will never do.
2. **Score that sample with the judge**, blind to the human label, with the
   grader model ID pinned and the rubric version recorded.
3. **Measure agreement against the human labels**, never against a second
   model. Report raw agreement and a chance-corrected statistic beside it -
   Cohen's kappa for a categorical verdict, Krippendorff's alpha or a rank
   correlation for an ordinal rubric. Raw agreement alone is not evidence: on
   a sample that is 90% pass, a judge answering pass every time scores 90%.
4. **Read every disagreement.** Each one says either that the judge is wrong
   or that the rubric is ambiguous, and a rubric two humans score differently
   is not repaired by a better judge.

What the agreement figure licenses:

| Chance-corrected agreement | What the judge's number is |
| --- | --- |
| not measured | unqualified; reported as unqualified, never quoted as a result |
| below 0.4 | unusable; fix the rubric or replace the judge with an executable check |
| 0.4 to 0.6 | usable only to compare two runs against the same rubric, never as an absolute score |
| above 0.6 | usable as an absolute score, reported with the agreement figure beside it |

Where those bands come from: they are the conventional reading of a
chance-corrected agreement statistic, not a figure measured here. No judge
this repository uses has been checked against them, and none can be yet - OMH's
own evaluation lanes score with deterministic predicates against frozen
corpora, so there is no grader model and no hand-labeled sample of its own to
measure against. Use them as the starting default and say that is what they
are when you quote one. The first row is a rule rather than a calibration and
holds whatever any measurement shows; the three numeric rows are the part that
moves. When you qualify your own judge, report the agreement figure with the
sample size beside it, and if that sample is too small to separate 0.4 from
0.6, that is the result rather than a reason to keep these numbers silently.

Every reported judge score carries four fields: the grader model ID, the
rubric version, the size of the sample agreement was measured on, and the
agreement value. A score missing any of them is prepared analysis, and the
report names the missing field instead of dropping the caveat.

## Boundary

A rubric, a loop design, or a judge score is prepared analysis. It is not
test evidence, review, CI, or merge evidence; a refined output is not a
verified output; and an improved score is a statement about the rubric, not
about the world.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall