dvy1987/agent-loom / .agents/skills/eval-judge
eval-judge
VerifiedScore LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 594 B |
| SKILL.md | skill md | 9.8 KB |
| references/examples.md | markdown | 2.8 KB |
| references/judge-calibration.md | markdown | 5.8 KB |
references/examples.md
View on GitHub# Eval Judge — Full Worked Examples
Skill: `eval-judge` | Enriched from SKILL.md (improve-skills pass, SKIP_RESEARCH).
## Example 1 — Documented workflow
**Input:** Score this API documentation response against our rubric
**Output:**
```
=== Evaluation Report ===
Target: API documentation response for /users endpoint
Rubric: docs/evals/2026-04-19-api-docs-rubric.md
Mode: Direct scoring
=== Hard Gates ===
| Gate | Verdict | Evidence |
|------|---------|----------|
| Format | PASS | Includes endpoint, method, params, response example |
| Safety | PASS | No internal URLs or credentials exposed |
=== Dimension Scores ===
### Accuracy — Score: 4/5 (confidence: 0.85)
Evidence: All parameter types correct. Response schema matches actual API.
Reasoning: One minor issue — rate limit header documented as X-Rate-Limit but actual is X-RateLimit-Remaining.
Improvement: Update rate limit header name to X-RateLimit-Remaining.
### Completeness — Score: 3/5 (confidence: 0.9)
Evidence: Missing error response examples for 401 and 429 status codes.
Reasoning: Core happy path documented but error handling incomplete — rubric requires all documented status codes.
Improvement: Add error response examples for 401 (unauthorized) and 429 (rate limited).
=== Summary ===
Hard gates: 2 pass, 0 fail
Dimensions: Accuracy 4/5, Completeness 3/5
Lowest-scoring: Completeness — missing error response examples
Overall verdict: PASS (all gates pass, quality improvements recommended)
```
## Example 2 — Step-by-step execution
**Input:** "Run `eval-judge` on [concrete task]"
**Agent actions:**
1. Gather Inputs
2. Choose Evaluation Mode
3. Direct Scoring
4. Pairwise Comparison (if two outputs)
5. Confidence Scoring
6. Internal Consistency Check (long-form outputs only)
7. Produce Evaluation Report
**Impact Report shape:**
```
Evaluation complete: [target]
Mode: [direct scoring / pairwise comparison]
Rubric used: [name/path]
Hard gates: [N] pass, [N] fail
Dimensions scored: [N]
Average confidence: [0-1]
Verdict: [PASS / FAIL / Winner A/B/TIE]
Key finding: [one-line summary]
```
## Example 3 — Gotcha application
**Input:** Task hits a non-obvious edge case
**Apply:**
- **Length ≠ quality.** Longer responses are systematically rated higher by LLM judges. Actively check: "Would a shorter version with the same content score equally?"
- **Confident tone ≠ accuracy.** Authoritative-sounding responses get higher scores even when wrong. Always verify factual claims against rubric criteria, not delivery style.
- **Chain-of-thought improves reliability 15-25%** but also increases token cost. Worth it for quality-critical evals; consider sampling for high-volume pipelines.
- In pairwise mode, if one output is much longer, the position swap is especially critical — length bias and position bias can compound.
---
See `SKILL.md` for hard rules and verification checklist.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |