Skip to content

richfrem/agent-plugins-skills / plugins/agent-orchestration/.apm/skills/evaluating-agent-outcomes

evaluating-agent-outcomes

Verified

Repository-independent evaluation of delegated agent work: assesses a delivery against a work contract (criterion-level pass/fail/unverified, failure conditions, budget, authorization), checks evidence validity, selects proportionate verification, measures cost with missing data kept unknown, classifies human interventions, and records and retrieves specific lessons. Use when orchestrating-agent-work needs findings after a worker returns, or to gate a benchmark before model calls. Not an orchestrator and never grants authority.

references/gates.md

View on GitHub
# Gates before and after model evaluation

| Situation | Gate (`eval_cli.py`) |
|---|---|
| Evaluator untested before responses | `fixtures`: run it on synthetic cases first |
| Output contract unproven | `pilot` on development items; pilot ids must not be reused as confirmatory items |
| Finished benchmark: what may it claim? | `validity` (legacy; assumes a relative-reduction claim): adherence < 0.8 -> MECHANICS_ONLY; no/floor/ceiling baseline -> UNSCORABLE. Prefer `assess_claim` |
| Claim-specific validity | `eval_core.assess_claim` with claim `{type: relative_reduction / absolute_difference / gain / descriptive_rate / adherence, baseline, treatment}`: a relative reduction from a zero baseline is UNDEFINED (value withheld) while absolute difference, descriptive rates (a zero is informative) and adherence stay reportable; gain needs headroom; low adherence caps every behavioral claim at MECHANICS_ONLY; descriptive rates are always returned; historical records are not rescored |
| Choosing the check | `select`: deterministic first; never shrink a preregistered requirement (checkpoint instead) |
| Human effort | `intervention`: required_authorization, necessary_clarification, execution_failure_correction, scope_change; only the third is a correction; an observed event is required |
| Same failure twice | `repair record`; later opportunities via `repair applied`; `repair report` gives recurrence over applicable opportunities |
| Cost | `invest`: execution, orchestration_review, retry_failed, verification, repair; unknown stays unknown |
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall