notque/vexjoy-agent / skills/meta/toolkit
toolkit
VerifiedToolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.
references/agent-comparison/benchmark-tasks.md
View on GitHub# Standard Benchmark Tasks
## Task Selection Principles
1. **Two-tier structure**: Simple tasks (algorithmic) + complex tasks (production)
2. **Simple tasks** test baseline capability. Both agents should perform identically
3. **Complex tasks** reveal real differences in edge case handling and production patterns
4. **Identical prompts**: Copy-paste the exact same description for both agents. No hints
## Simple Task: Advent of Code
Use Day 1-6 of any year. These have:
- Clear input/output specification
- Testable with provided examples
- No ambiguity in requirements
- No domain-specific knowledge needed
### Example Prompt (AoC-style)
```
Solve this algorithm problem in Go.
Problem: Given a list of integers, find the two entries that sum to 2020
and return their product.
Input format: One integer per line
Example:
1721
979
366
299
675
1456
Expected output for example: 514579 (1721 * 299)
Requirements:
1. Create solution in main.go
2. Write comprehensive tests in main_test.go
3. Include the provided example as a test case
4. Handle edge cases appropriately
Save files to: benchmark/{task-name}/{agent-variant}/
```
## Complex Task: Worker Pool
Production-style concurrent processing. Reveals differences in:
- Goroutine lifecycle management
- Graceful shutdown patterns
- Error handling under concurrency
- Race condition prevention
### Prompt
```
Build a production-ready rate-limited worker pool in Go.
Requirements:
- Configurable number of workers
- Configurable queue size with backpressure options
- Token bucket rate limiting
- Graceful shutdown with timeout
- Context cancellation support
- Metrics (jobs processed, failed, queue depth)
- Health check endpoint data
- Panic recovery in workers
Include comprehensive tests covering:
- Basic job processing
- Rate limiting behavior
- Backpressure (blocking vs error)
- Graceful shutdown
- Context cancellation
- Concurrent access safety (use -race flag)
Save to: benchmark/workerpool/{agent-variant}/
```
## Complex Task: LRU Cache with TTL
Tests generic programming, background goroutines, and semantic correctness. Known to expose bugs around zero-value handling.
### Prompt
```
Build a generic LRU cache with TTL support in Go.
Requirements:
- Generic types for key and value
- Configurable max size
- Per-item TTL support
- Background cleanup goroutine
- Thread-safe operations
- Methods: Get, Set, Delete, Clear, Len, Stats
Include comprehensive tests with race detection.
Save to: benchmark/cache/{agent-variant}/
```
### Known Bug Patterns (from December 2024 testing)
These bugs appeared in 2-3 out of 4 tested agents:
| Bug | Frequency | Production Impact |
|-----|-----------|-------------------|
| TTL=0 treated as "expire immediately" | 2/4 agents | Items vanish on insert |
| Clear() returns nothing | 3/4 agents | Cannot track cache evictions |
| No WaitGroup for background goroutine | 3/4 agents | Goroutine leak on shutdown |
| Delete() doesn't return existence | 2/4 agents | Cannot do conditional logic |
## Complex Task: HTTP Service
Tests standard patterns: middleware, routing, logging, error handling.
### Prompt
```
Build an HTTP service in Go with the following endpoints:
- POST /items - Create item
- GET /items/:id - Get item by ID
- GET /items - List items with pagination
- DELETE /items/:id - Delete item
Requirements:
- In-memory storage (no database)
- Request logging middleware
- Request ID middleware
- Graceful shutdown
- Health check endpoint at /health
- Structured JSON error responses
- Input validation
Include comprehensive tests for all endpoints and middleware.
Save to: benchmark/httpservice/{agent-variant}/
```
## Benchmark Directory Structure
```
benchmark/
{task-name}/
full/
main.go
main_test.go
go.mod
compact/
main.go
main_test.go
go.mod
```
## Running Benchmarks
### Step 1: Create directories
```bash
mkdir -p benchmark/{task-name}/{full,compact}
```
### Step 2: Run agents in parallel
Use Task tool to spawn both agents simultaneously for fair timing:
```
Task(prompt="[task prompt]\nSave to: benchmark/{task}/full/", subagent_type="{full-agent}")
Task(prompt="[task prompt]\nSave to: benchmark/{task}/compact/", subagent_type="{compact-agent}")
```
### Step 3: Validate with race detection
```bash
cd benchmark/{task-name}/full && go test -race -v
cd benchmark/{task-name}/compact && go test -race -v
```
### Step 4: Run comparison analysis
```bash
# TODO: scripts/compare.py not yet implemented
# Manual alternative: compare outputs side-by-side using diff
diff benchmark/{task-name}/full/ benchmark/{task-name}/compact/
```
## Optimization Loop Task Format
The current optimization loop is for frontmatter-description and routing-trigger quality. It does not run full code-generation benchmarks. Use Phase 5 with trigger-rate eval tasks, then use Phases 1-4 for full agent benchmarking.
### Supported Task File Schemas
Flat list with optional split markers:
```json
{
"tasks": [
{
"name": "go-patterns-positive",
"split": "train",
"complexity": "complex",
"query": "write table-driven tests for a Go parser with subtests and helpers",
"should_trigger": true
},
{
"name": "kubernetes-negative",
"split": "test",
"complexity": "complex",
"query": "debug a kubernetes pod stuck in CrashLoopBackOff",
"should_trigger": false
}
]
}
```
Explicit top-level train/test sets:
```json
{
"train": [
{
"name": "positive-1",
"query": "write Go benchmarks and race tests for a worker pool",
"should_trigger": true
}
],
"test": [
{
"name": "negative-1",
"query": "design a PostgreSQL indexing strategy",
"should_trigger": false
}
]
}
```
### Required Fields
- `query`: the prompt used to test routing behavior
- `should_trigger`: expected boolean outcome for the target description
### Optional Fields
- `name`: human-readable label shown in reports
- `split`: `train` or `test` when using a flat `tasks` list
- `complexity`: used for stratified splitting when no explicit split is provided
### Split Strategy
- `train` tasks are used during each optimization iteration.
- `test` tasks are held out and checked every 5 iterations for Goodhart divergence.
- If no split markers are present, the loop performs a reproducible random split with seed `42`, stratified by `complexity`.
### Task Selection Principles for Optimization
1. Cover both positive and negative routing examples. A description that only improves recall while tanking precision is not an improvement.
2. Put at least one out-of-domain prompt in the held-out set. This catches overfitting where the description starts matching benchmark phrasing instead of the real scope.
3. Use realistic user wording, not only canonical trigger phrases. Optimization on synthetic wording alone produces brittle routing behavior.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |