Arnie016/codex-prompt-templates / skills/auto-skill-build-agent-reliability-loop
auto-skill-build-agent-reliability-loop
VerifiedDesign production agent reliability loops across traces, evals, guardrails, gateways, model budgets, and shadow tests. Use when shipping AI/agent workflows that need simulate/evaluate/protect/monitor/optimize structure, gateway policy, eval suites, guardrail rollout, or trace-to-regression feedback. Skip when the task only needs a local Codex run ledger; use `$agent-observability` instead.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.9 KB |
SKILL.md
View on GitHub---
name: auto-skill-build-agent-reliability-loop
description: >
Design production agent reliability loops across traces, evals, guardrails,
gateways, model budgets, and shadow tests. Use when shipping AI/agent
workflows that need simulate/evaluate/protect/monitor/optimize structure,
gateway policy, eval suites, guardrail rollout, or trace-to-regression
feedback. Skip when the task only needs a local Codex run ledger; use
`$agent-observability` instead.
---
# Agent Reliability Loop
Generated by: Codex Supercharge maintenance automation.
Goal: turn an agent workflow into a closed loop where production traces become
eval cases, eval failures become guardrails or fixes, and gateway policy keeps
cost, routing, and tool risk bounded.
## Skip When
- The user only wants local Codex run/cost tracking.
- The work is a single benchmark or experiment with no production surface.
- The task is only MCP conformance; use `$auto-skill-build-mcp-conformance-harness`.
## Workflow
1. **Map the surface**: list user journeys, models, tools, data classes,
side effects, risk levels, and quality/cost/latency targets.
2. **Instrument first**: require request IDs, trace/span IDs, session IDs,
model/provider, token/cost, cache, fallback, guardrail, and tool metadata.
3. **Build eval bundles**: create golden cases for task success, prompt
conformance, tool correctness, unsafe requests, PII/secrets, latency, and
cost. Keep judge prompts and heuristic checks versioned.
4. **Set gateway policy**: define provider mapping, model fallbacks, cache
rules, virtual keys, budgets, rate limits, privacy redaction, and audit logs.
5. **Stage guardrails**: run new pre/post checks in log or monitor mode first,
then enforce only after false positives and fail-open/fail-closed behavior
are explicit.
6. **Shadow safely**: mirror sampled traffic to candidate models or prompts
only when shadow calls cannot trigger external side effects.
7. **Close the loop**: cluster failed traces, match nearest successful traces,
add representative failures to evals, patch prompts/tools/policies, then
rerun the bundle before release.
## Commands
```bash
rg -n "trace|span|cost|tokens|guardrail|fallback|budget|rate limit|request_id" .
rg -n "eval|rubric|judge|golden|dataset|experiment|shadow|canary" .
rg -n "tool|mcp|side effect|webhook|shell|filesystem|credential" .
```
## Output
```md
# Agent Reliability Plan
## Surface
## Instrumentation
## Eval Bundle
## Gateway Policy
## Guardrail Rollout
## Shadow Or Canary Plan
## Trace-To-Regression Loop
## Risks And Trust Notes
## Validation
```
## Validation
- Every production route has at least one trace and one regression case.
- Every blocking guardrail has a false-positive review path.
- Every budget/rate limit has an owner and an alert threshold.
- Shadow/canary traffic cannot write to tools, accounts, payments, or user data.
Read `references/future-agi-agent-reliability-loop.md` for the source-backed
pattern and risk notes.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |