Skip to content

existential-birds/beagle / plugins/beagle-analysis/skills/llm-judge

llm-judge

Verified

Use when comparing two or more code implementations against a spec or requirements doc. Triggers on "which repo is better", "compare these implementations", "evaluate both solutions", "rank these codebases", or "judge which approach wins". Also covers choosing between competing PRs or vendor submissions solving the same problem. Does NOT review a single codebase for quality — use code review skills instead. Does NOT evaluate strategy docs — use strategy-review. Requires a spec file and 2+ repo paths.

FileKindSize
SKILL.mdskill md10.7 KB
references/fact-schema.mdmarkdown4.3 KB
references/judge-agents.mdmarkdown2.7 KB
references/repo-agent.mdmarkdown3.1 KB
references/scoring-rubrics.mdmarkdown4.1 KB

references/scoring-rubrics.md

View on GitHub
# Scoring Rubrics

Detailed rubrics for each of the 5 judging dimensions. Judges use these to assign consistent 1-5 scores.

## General Scoring Scale

| Score | Meaning | General Criteria |
|-------|---------|------------------|
| 5 | Excellent | Exceeds expectations, best practices throughout |
| 4 | Good | Meets all requirements, minor issues only |
| 3 | Average | Functional but notable gaps or issues |
| 2 | Below Average | Significant issues affecting quality |
| 1 | Poor | Fails to meet basic requirements |

---

## Functionality (30% weight)

Evaluates whether the implementation meets the spec requirements and works correctly.

| Score | Criteria |
|-------|----------|
| 5 | All spec requirements implemented. All tests pass. No obvious bugs. |
| 4 | All requirements implemented. Tests pass with minor failures (< 5%). Edge cases may be missing. |
| 3 | Most requirements implemented (> 75%). Some test failures. Core functionality works. |
| 2 | Partial implementation (50-75%). Significant test failures. Core features have bugs. |
| 1 | Minimal implementation (< 50%). Tests fail or don't exist. Core functionality broken. |

**Key Evidence:**
- `functionality.implemented` vs `functionality.spec_requirements`
- `functionality.test_results.passed` vs `functionality.test_results.failed`
- `functionality.missing` and `functionality.partially_implemented`

---

## Security (25% weight)

Evaluates security posture and absence of vulnerabilities.

| Score | Criteria |
|-------|----------|
| 5 | No security findings. Positive security patterns present. OWASP Top 10 addressed. |
| 4 | No high-severity findings. 1-2 low/medium issues. Good security hygiene. |
| 3 | 1-2 medium-severity issues OR 3+ low-severity. Basic security present. |
| 2 | 1+ high-severity issue OR 3+ medium. Security gaps evident. |
| 1 | Multiple high-severity issues. Critical vulnerabilities. No security consideration. |

**Severity Weights:**
- High: SQL injection, command injection, auth bypass, secrets in code
- Medium: XSS, CSRF, insecure deserialization, missing input validation
- Low: Information disclosure, verbose errors, missing security headers

**Key Evidence:**
- `security.findings` (count and severity)
- `security.patterns_observed`

---

## Test Quality (20% weight)

Evaluates test coverage, DRY adherence, and testing practices.

| Score | Criteria |
|-------|----------|
| 5 | High coverage. No DRY violations. Good mock boundaries. Tests are maintainable. |
| 4 | Moderate-high coverage. Minor DRY issues (1-2). Good testing practices. |
| 3 | Moderate coverage. Some DRY violations (3-5). Acceptable mocking. |
| 2 | Low coverage. Significant DRY violations. Poor mock boundaries. |
| 1 | Minimal/no tests. Severe DRY problems. Tests don't follow best practices. |

**Key Evidence:**
- `tests.coverage_estimate`
- `tests.dry_violations` (count)
- `tests.mocking_approach`
- `tests.test_count` relative to codebase size

---

## Overengineering (15% weight)

Evaluates simplicity and absence of unnecessary complexity.

| Score | Criteria |
|-------|----------|
| 5 | Clean, simple code. No unnecessary abstractions. YAGNI followed. |
| 4 | Mostly simple. 1-2 minor over-abstractions. Code is readable. |
| 3 | Some complexity. 3-5 abstraction issues. Config complexity medium. |
| 2 | Significant over-engineering. 6+ abstraction issues. Unnecessary patterns. |
| 1 | Severely over-engineered. Abstractions everywhere. Simple tasks made complex. |

**Key Evidence:**
- `overengineering.abstractions` (count)
- `overengineering.defensive_code` (count)
- `overengineering.config_complexity`

---

## Dead Code (10% weight)

Evaluates cleanliness and absence of unused/obsolete code.

| Score | Criteria |
|-------|----------|
| 5 | No dead code. No TODOs. Clean codebase. |
| 4 | 1-3 minor issues (unused imports). No significant dead code. |
| 3 | 4-6 issues. Some unused functions or TODOs. |
| 2 | 7-10 issues. Unused functions/classes. Multiple TODOs. |
| 1 | 10+ issues. Significant dead code. Many TODOs/commented blocks. |

**Key Evidence:**
- `dead_code.unused_imports` (count)
- `dead_code.unused_functions` (count)
- `dead_code.todo_comments`
- `dead_code.commented_code_blocks`
rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).Stars: 198kUpdated Claude CodeCodexUnverifiedInstall
doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks.Stars: 198kUpdated Claude CodeCodexVerifiedInstall