TheGoat395/Codex-Skills / skills/agent-evaluation-operations
agent-evaluation-operations
VerifiedEvaluate agent and skill behavior or routing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.7 KB |
| agents/openai.yaml | config | 256 B |
| references/fixture-execution.md | markdown | 2.0 KB |
| references/fixtures/draft-send.json | config | 351 B |
| references/regression-corpus.json | config | 19.9 KB |
| references/skill-behavior-evaluation.md | markdown | 3.1 KB |
| scripts/analyze_skill_collisions.pyexecutable | script | 5.0 KB |
| scripts/observe_draft_fixture.py | script | 1.1 KB |
| scripts/test_observe_draft_fixture.py | script | 841 B |
| scripts/validate_regression_corpus.pyexecutable | script | 4.6 KB |
references/fixture-execution.md
View on GitHub# Fixture execution and observer boundary
The regression corpus is a versioned scenario specification. Its structural validator does not run a model, and prose allowed-resources lists are not executable fixtures. Before a live comparison, bind each selected case to an actual bounded fixture, harness adapter and observable scorer. Unbound cases remain unexecuted; never report a suite pass from their schema.
The bundled `fixtures/draft-send.json` is a concrete synthetic dossier for the draft-versus-send boundary. A local harness can expose only a temporary draft folder and an outbox event log, run a candidate against the draft request, then call `scripts/observe_draft_fixture.py <fixture-root>`. The observer checks actual draft bytes and outbox events. Do not let candidate self-reported status substitute for harness-owned events. A trace-owning adapter must keep its outbox ledger outside the candidate's write permissions when used for a real model evaluation.
The scorer is deliberately narrow: it does not prove factual quality for arbitrary email, credential safety, live delivery or all corpus cases. `scripts/test_observe_draft_fixture.py` tests it with safe local fixtures and an adversarial send; this is executable helper testing, not a model evaluation.
For other scenarios, create the case-specific project files, rendered inputs, mock service state or supplied source dossiers before dispatch. If a separately installed `honesty` skill supplies comparison dossiers, inspect their scope before use; it is not bundled here. Otherwise construct a small synthetic dossier from the selected corpus case with explicit facts, expected output and prohibited effects. Do not invent external observations to populate it. Hold prompts, fixture facts, model settings and tools constant between baseline and treatment, apart from the intended change. Capture immutable run IDs, exact candidate version and independently observed effects; score first attempts before retries and retain failures. If no adapter is available, report a semantic walkthrough and its limit.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |