richfrem/agent-plugins-skills / plugins/agent-orchestration/.apm/skills/evaluating-agent-outcomes
evaluating-agent-outcomes
VerifiedRepository-independent evaluation of delegated agent work: assesses a delivery against a work contract (criterion-level pass/fail/unverified, failure conditions, budget, authorization), checks evidence validity, selects proportionate verification, measures cost with missing data kept unknown, classifies human interventions, and records and retrieves specific lessons. Use when orchestrating-agent-work needs findings after a worker returns, or to gate a benchmark before model calls. Not an orchestrator and never grants authority.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.6 KB |
| assets/rubrics/pr-review.json | config | 194 B |
| evals/evals.json | config | 724 B |
| references/gates.md | markdown | 1.5 KB |
| references/scoring-and-accounting.md | markdown | 1.9 KB |
| references/work-contract.md | markdown | 2.7 KB |
| scripts/accounting.py | script | 17.2 KB |
| scripts/eval_cli.py | script | 3.5 KB |
| scripts/eval_core.py | script | 16.3 KB |
| scripts/lessons.py | script | 5.6 KB |
| scripts/outcome_eval.py | script | 8.6 KB |
| scripts/scoring.py | script | 4.7 KB |
| scripts/usage_measure.py | script | 16.1 KB |
| tests/fixtures/agy_usage.json | config | 2.4 KB |
| tests/fixtures/claude_usage.txt | markdown | 718 B |
| tests/fixtures/claude_usage_resetless.txt | markdown | 215 B |
| tests/fixtures/sanctuary-contribution-v1.json | config | 794 B |
| tests/test_eval.py | script | 8.0 KB |
| tests/test_scoring_accounting.py | script | 5.7 KB |
references/scoring-and-accounting.md
View on GitHub# Scoring, accounting and usage measurement (generic, single implementation)
| Module | Responsibility | Caller supplies |
|---|---|---|
| `scoring.py` | Profile-driven scoring: validate ratings (no clamping), raw sum, `raw_fraction` or `weighted` headline, cost-adjusted ROI, optional bloat rule, deterministic event-log projection (`fold_assessment`, `rebuild_projection`) | A profile (dimensions, scale, weights, headline mode, bloat rule) |
| `accounting.py` | Model cost tiers and the historical substring lookup (default standard 1.0), usage-ledger fold (`fold_usage`, 30 recent), turn-budget check | Budget profiles (policy), ledger path |
| `usage_measure.py` | Zero-token vendor usage snapshots and diffs (claude `/usage`, agy JSON, codex rollouts), transcript token readers, secret scrubbing, catalog-priced `cost_usd` (None when a price is absent, never guessed) | Where to store snapshots, floors, policy |
| `outcome_eval.py` | Contract assessment; optional rubric scores come from `scoring.py` | Contract, evidence |
## Profiles
`assets/rubrics/pr-review.json` is an ordinary-work example (`weighted`: weights 3/2/1/1, headline `sum(w*v/max)/sum(w)`).
A project can bring its own: e.g. Sanctuary's five 0-3 dimensions use `fitness: raw_fraction` (headline `round(raw/15, 3)`) and a bloat rule
(model weight >= 3.0 with low token economy or outcome). The two profiles differ BY DESIGN: weights and the headline definition. The same
mechanism computes both (tests: `test_scoring_accounting.py`, fixture `tests/fixtures/sanctuary-contribution-v1.json`); a project's historical
scores stay valid because its profile reproduces its original arithmetic. Scores never alter `complete` or `integration_allowed`.
## What stays with the consumer
Budget policy (subscription profiles, soft limits), headroom floors and static ceilings, any predictor, attribution to its own work items,
storage locations and event names, dashboards and text output.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |