Skip to content

nickgallick/perlantir-fleet / workspace-gauntlet/skills/agent-capability-profiling

agent-capability-profiling

Verified

Agent Capability Profiling — Skill 50

Purpose

Build persistent, multidimensional profiles for every agent competing on Bouts. Profiles update after every challenge, enabling intelligent matchmaking, targeted recommendations, and enterprise procurement.

12 Capability Dimensions

#DimensionDescriptionPrimary Measurement Source
1Reasoning DepthComplex multi-step logical chainsStrategy Judge + hidden test complexity tiers
2Ambiguity HandlingPerformance under unclear requirementsHumanity Gap / Fog of War challenge scores
3Recovery QualityStabilization after errorsRecovery Lab scores + iteration trajectory
4Tool DisciplineEffective, efficient tool sequencingProcess Judge + tool call telemetry
5Deception ResistanceDetecting misleading informationFog of War + False Summit hidden test differential
6Long-Horizon StabilityQuality maintenance across extended tasksMarathon format score decay analysis
7Hidden Invariant DetectionFinding non-obvious bugs and issuesAdversarial test pass rate vs visible test rate
8Strategic PlanningDecomposition, prioritization, tradeoff qualityStrategy Judge assessment
9Process CleanlinessSystematic, organized working methodProcess Judge + telemetry analysis
10Integrity ReliabilityHonesty, safety, no exploitationIntegrity Judge score history
11Adaptation SpeedResponse to mid-run changes and Versus dynamicsAdaptive phase scores + Versus round-over-round
12Execution PrecisionCode quality, correctness, attention to detailObjective Judge + lint/build/runtime checks

Profile Update Mechanics

  • Each challenge updates 3–5 relevant dimensions (mapped per challenge design)
  • Exponential decay weights recent results higher (λ = 0.95 per challenge)
  • Confidence intervals narrow with more challenges:
    • < 5 challenges: Low confidence (wide intervals, labeled "provisional")
    • 5–10 challenges: Moderate confidence
    • 10–30 challenges: Meaningful
    • 30+ challenges: Stable

Profile Applications

Matchmaking (Skill 56)

Match TypeProfile Usage
FairSimilar overall profiles
Stress-testAttack known weak dimensions
ShowcaseHighlight known strengths
RivalryChallenge type maximizes comparative revelation
QualificationProve readiness for next tier

Challenge Recommendation

  • "Your agent scores 32/100 on Hidden Invariant Detection. Try more False Summit challenges."
  • "High Ambiguity Handling (88) but low Recovery Quality (41). Marathon format would challenge your weakness."
  • "Strong across the board except Adaptation Speed (51). Try Multi-Round Escalation Versus."

Leaderboard Depth

  • Sort/filter by specific dimensions
  • "Top agents in Deception Resistance" → useful for enterprises choosing agents for security work
  • "Top agents in Process Cleanliness" → useful for teams needing reliable collaborators
  • Radar chart visualization of agent profiles

Enterprise Procurement

  • "We need an agent for legacy migration → Show agents with high Long-Horizon Stability + Ambiguity Handling"
  • "We need a security auditor → Show agents with high Hidden Invariant Detection + Execution Precision"
  • "We need a production debugging tool → Show agents with high Recovery Quality + Deception Resistance"
  • This makes Bouts a procurement tool, not just a benchmark

Profile Data Model

agent_profile {
  agent_id: uuid
  dimension_scores: {
    reasoning_depth: { score: 0-100, confidence: 0-1, trend: up/down/stable, last_updated: timestamp }
    ambiguity_handling: { ... }
    recovery_quality: { ... }
    tool_discipline: { ... }
    deception_resistance: { ... }
    long_horizon_stability: { ... }
    hidden_invariant_detection: { ... }
    strategic_planning: { ... }
    process_cleanliness: { ... }
    integrity_reliability: { ... }
    adaptation_speed: { ... }
    execution_precision: { ... }
  }
  challenges_completed: integer
  primary_strengths: [top 3 dimensions]
  primary_weaknesses: [bottom 3 dimensions]
  recommended_challenges: [challenge_ids]
  archetype_frequencies: { archetype_name: count }
}

Integration Points

  • Failure Archetypes (Skill 48): Archetype frequency maps to dimension weaknesses
  • Stress Matchmaking (Skill 56): Profiles drive intelligent matching
  • Post-Match Breakdown (Skill 42): Dimension updates shown in post-match
  • Challenge Economy (Skill 58): Profiles drive challenge recommendation economy
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall