Skip to content

mouadja02/skills / skills/agent-eval/mcp-agent-evaluation

mcp-agent-evaluation

Verified

Use when evaluating MCP-enabled agents, tool-using LLM systems, multi-server workflows, tool-selection quality, trajectory quality, task completion, or MCP security regressions.

MCP Agent Evaluation

Use this skill to design practical evaluations for agents that discover, select, and call MCP tools. Focus on whether the agent can solve real tasks with live or simulated toolchains, not only whether it can call a single tool correctly.

Activation Checklist

Activate when the user asks to:

  • Benchmark an MCP client, coding agent, or multi-tool agent
  • Compare models on tool selection, planning, or task completion
  • Build evaluation tasks from real MCP servers
  • Diagnose failed tool-use trajectories
  • Add security or prompt-injection tests around MCP tool metadata and outputs

Evaluation Model

Evaluate at four levels:

  1. Tool discovery - the agent finds relevant tools without being handed exact tool names.
  2. Schema and parameter use - the agent fills arguments correctly, handles optional fields, and respects tool constraints.
  3. Trajectory quality - the agent sequences calls coherently, uses intermediate outputs, and recovers from tool errors.
  4. Outcome quality - the final answer is correct, grounded in tool results, and safe to deliver.

Add a fifth track for security when untrusted tool metadata, retrieved content, or tool outputs can influence future calls.

Workflow

  1. Inventory the environment

    • List MCP servers, tools, schemas, authentication boundaries, side effects, and domain coverage.
    • Mark tools as read-only, write-capable, privileged, networked, or user-data-sensitive.
    • Record which tools are complementary and should be used together.
  2. Create task families

    • Fuzzy lookup: user describes the goal without naming the tool.
    • Multi-hop: success requires two or more dependent calls.
    • Cross-domain: success requires tools from different servers.
    • Error recovery: one call fails or returns incomplete data.
    • Grounding: answer must cite or summarize intermediate tool outputs.
    • Security: malicious descriptions, injected tool output, overbroad scopes, and unsafe write attempts.
  3. Define scoring before running agents

    • Use binary pass/fail for critical outcomes.
    • Use graded rubrics for trajectory, grounding, and recovery.
    • Separate model failures from harness failures and unavailable tools.
    • Store full trajectories, not just final answers.
  4. Run minimal baselines

    • Single explicit tool task.
    • Single server multi-step task.
    • Multi-server task with fuzzy instructions.
    • Security canary task where the correct behavior is refusal, confirmation, or sandboxing.
  5. Analyze failure modes

    • Wrong tool chosen
    • Correct tool with wrong arguments
    • Correct calls but poor state tracking
    • Ungrounded final answer
    • Unsafe trust in tool metadata or output
    • Excessive retries or runaway tool loops

Rubric

DimensionPass SignalFailure Signal
Tool retrievalRelevant tools selected from vague requestAgent ignores available tools or picks lexical near misses
Schema useRequired and optional fields are validHallucinated fields, invalid enum values, missing constraints
PlanningCalls are ordered by dependencyCalls are redundant, circular, or skip prerequisite data
GroundingFinal answer reflects tool outputsFinal answer invents results or ignores returned evidence
RecoveryAgent adapts to errors or ambiguityAgent repeats the same failed call
SafetySide effects are gated and instructions are isolatedTool output or metadata overrides user/system intent

Helper Script

Use mcp_eval_plan.py to turn a server/task inventory into a Markdown evaluation plan:

python scripts/mcp_eval_plan.py inventory.json

The input may contain domains, servers, task_families, and risks. Missing fields use conservative defaults.

References

Read benchmark-map.md before designing a new benchmark suite or choosing which public MCP benchmark to mirror.

External grounding:

rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall