Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/agent-tool-replay-test

agent-tool-replay-test

Verified

Replay recorded agent tool/LLM trajectories against deterministic fixtures to test argument validation, errors, idempotency, and side-effect boundaries—separate from live tool-selection quality.

Agent Tool Replay Test AI Skill Guide (Claude)

Overview & Engine Architecture

Live agents are expensive and flaky. Record/replay at the SDK or tool boundary gives deterministic CI: assert validation, error handling, and “must not call” policies without paying for tokens. Replay does not prove the model still chooses the right tools—that needs live/eval gates (@prompt-regression-gate).

Claude operates as a Principal Agent Test Architect for cassettes, contract assertions, clock/ID freezing, and unknown-outcome handling after timeouts.

┌─────────────────────────────────────────────────────────────┐
│  RECORD (once): LLM decisions ± tool results → cassette     │
│  REPLAY (CI): inject recordings; optionally run real tools  │
│  ASSERT: schemas, sequences, deny-lists, idempotency        │
└─────────────────────────────────────────────────────────────┘

Scope

  • Read tool contracts: read-only vs mutating.
  • Use project test runner + mocks; no production networks in replay mode.
  • Sanitize cassettes (no credentials)—@ai-pii-redaction-review.
  • Label coverage: replay contract ≠ e2e model behavior.

Fixture matrix (minimum)

CaseExpect
SuccessValid args → OK result
TimeoutBounded retry policy; unknown outcome path
Permission deniedNo side effect; surfaced error
Malformed tool JSONReject via @llm-json-contract-check
Partial failureCompensating / reconcile logic
Unknown fields / bad IDsValidation reject before dispatch
Mutating callAuthZ + idempotency key before effect
Policy denyTool absent from allow-list never invoked

Determinism rules

  • Freeze clocks/randomness; stable paths in prompts (no uuid4() in recorded args).
  • Modes: record | replay (CI; miss = fail, no network) | passthrough.
  • Prefer SDK-boundary recorders (OpenAI/Anthropic patch) over brittle raw HTTP VCR for agents.
  • Two styles: (A) stub tool results entirely; (B) replay LLM decisions but execute tools against safe stubs—pick explicitly.

Timeout / idempotency (critical)

After a timeout on charge/send_email, state is unknown. Blind replay can duplicate. Require: idempotency keys, reconcile-before-retry, or human handoff (@ai-human-handoff-contract).


Assertions to ship

  • Tool sequence / called_with schemas
  • Forbidden tools never called
  • Budget/step caps
  • Cassette diff on prompt drift (fail or re-record deliberately)

Deliverable

Fixtures + traces + “must not occur” assertions + explicit statement that selection quality is out of scope for pure replay.

Related skills

@llm-json-contract-check, @prompt-regression-gate, @agent-injection-boundary-test, @llm-cost-latency-benchmark

Agent Operational Directive

MANDATORY: Keep replay offline in CI. Sanitize cassettes. Test validation and side-effect gates. Never blind-retry mutating calls after unknown outcomes. Separate replay coverage from model tool-selection evals.

Sources

Cassette/agentverify/pytest-agentcontract/langchain-replay patterns; industry emphasis on SDK-level record/replay and idempotent mutating tools.

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall