Skip to content

richfrem/agent-plugins-skills / plugins/agent-orchestration/.apm/skills/evaluating-agent-outcomes

evaluating-agent-outcomes

Verified

Repository-independent evaluation of delegated agent work: assesses a delivery against a work contract (criterion-level pass/fail/unverified, failure conditions, budget, authorization), checks evidence validity, selects proportionate verification, measures cost with missing data kept unknown, classifies human interventions, and records and retrieves specific lessons. Use when orchestrating-agent-work needs findings after a worker returns, or to gate a benchmark before model calls. Not an orchestrator and never grants authority.

references/work-contract.md

View on GitHub
# Work contract, findings and records (v1)

Contract (JSON; caller-defined `id`/`type`, no mission schema needed):

```json
{"id": "t1", "type": "task|issue|pr|experiment|...", "goal": "...",
 "scope": {"paths": ["files or dirs that may change"]},
 "acceptance": [{"id": "A1", "text": "...", "required": true,
                 "check": {"kind": "command|file_exists|file_contains|manual", "argv": ["..."], "expect_exit": 0, "expect_stdout_contains": ["..."]}}],
 "failure_conditions": [{"id": "F1", "text": "...", "check": {"kind": "file_exists", "path": "ERROR.txt"}}],
 "budget": {"max_attempts": 3, "max_tokens": 50000, "per_attempt_token_reserve": 8000},
 "authorization": {"may_merge": false},
 "rubric": {"id": "optional", "dimensions": {"correctness": {"max": 3, "weight": 3}}}}
```

Checks run with cwd confined to the root. A passing failure-condition check means the failure occurred. A `manual` criterion is `unverified` until `evidence.manual_confirmations[id]` cites a reference.

Findings (`outcome_eval.assess`): `criteria[]` (result pass|fail|unverified, detail, evidence_ref), `failure_hits`, `incomplete`, `blockers`, `limitations`, `usage` (tokens or null), `complete`, optional `scores` (rubric id, normalized value, problems), `integration_allowed` (requires complete AND the caller's authorization; scores irrelevant), `recommended` (accept | repair | blocked | escalate | fix_contract), `repair_instructions`.

Records (`eval_core`, JSONL, `v:1`, `kind`): `lesson`, `lesson_use`, `intervention`, `repair`, `repair_application`. Legacy `event_type` lines are still read. Store must resolve inside the declared root. Nested workers: attribute tokens once, to the worker's own unit. API-equivalent price is never mixed with subscription token usage.

Lesson fields: trigger, failure, evidence, cause (known|suspected), action, verification, limits, tags. `find` returns up to 3 compact matches. A lesson earns no credit until `lesson_use` records an outcome (helped|no_effect|harmful).

## Budget (validated by `accounting.validate_budget`; per work item)
`max_attempts` required positive integer (booleans, floats, strings, zero, negatives rejected); `max_tokens` and `per_attempt_token_reserve` optional positive
integers (reserve <= ceiling, needs a ceiling); unknown keys rejected. `accounting.summarize_attempts` folds an append-only attempt ledger (last record per
attempt id wins; each backend attempt counted once; unknown usage stays unknown; running attempts count as unknown) and `accounting.admit` decides whether one
more execution attempt may launch. Preflight refusals and evaluation calls are tracked separately from execution attempts. Totals are lower bounds whenever
anything is unknown. Token limits are admission control: no backend guarantees a hard cap.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall