Skip to content

existential-birds/beagle / plugins/beagle-analysis/skills/llm-judge

llm-judge

Verified

Use when comparing two or more code implementations against a spec or requirements doc. Triggers on "which repo is better", "compare these implementations", "evaluate both solutions", "rank these codebases", or "judge which approach wins". Also covers choosing between competing PRs or vendor submissions solving the same problem. Does NOT review a single codebase for quality — use code review skills instead. Does NOT evaluate strategy docs — use strategy-review. Requires a spec file and 2+ repo paths.

FileKindSize
SKILL.mdskill md10.7 KB
references/fact-schema.mdmarkdown4.3 KB
references/judge-agents.mdmarkdown2.7 KB
references/repo-agent.mdmarkdown3.1 KB
references/scoring-rubrics.mdmarkdown4.1 KB

references/repo-agent.md

View on GitHub
# Repo Agent Instructions

Instructions for Phase 1 agents that gather facts from a single repository.

## Role

You are a fact-gathering agent. Your job is to explore a repository and extract structured facts WITHOUT making judgments or assigning scores. Scoring happens in Phase 2 by separate judge agents.

## Inputs You Receive

1. **Spec Document**: The requirements/plan that was given to the LLM to implement
2. **Repo Path**: Absolute path to the repository you're analyzing
3. **Repo Label**: Display name for this repo (e.g., "Claude", "GPT-4")
4. **Branch Info**: Which branch to compare (default: current vs main)

## Your Task

Produce a JSON object following the schema in [fact-schema.md](fact-schema.md).

## Step-by-Step Process

### 1. Gather Git Info

```bash
# Get branch name
git -C $REPO_PATH rev-parse --abbrev-ref HEAD

# Get diff stats
git -C $REPO_PATH diff --stat main...HEAD

# Count files changed
git -C $REPO_PATH diff --name-only main...HEAD | wc -l
```

### 2. Analyze Functionality

1. Read the spec document carefully
2. Extract discrete requirements as a list
3. Explore the codebase to determine which requirements are implemented
4. Run tests if available:

```bash
# Detect and run tests
cd $REPO_PATH

# Python
if [ -f pytest.ini ] || [ -f pyproject.toml ] || [ -d tests ]; then
  pytest --tb=short 2>&1
fi

# JavaScript/TypeScript
if [ -f package.json ]; then
  npm test 2>&1 || yarn test 2>&1
fi

# Go
if [ -f go.mod ]; then
  go test ./... 2>&1
fi
```

### 3. Analyze Security

Look for common vulnerabilities:
- SQL injection (string concatenation in queries)
- Command injection (unsanitized shell commands)
- XSS (unsanitized user input in HTML)
- Hardcoded secrets (API keys, passwords)
- Missing input validation
- Insecure deserialization

Also note positive patterns:
- Input validation present
- Parameterized queries
- Authentication checks
- Rate limiting

### 4. Analyze Tests

- Count test files and test functions
- Look for DRY violations (repeated setup code)
- Assess mocking strategy
- Estimate coverage (file count ratio, critical paths tested)

### 5. Analyze Overengineering

Use patterns from the **llm-artifacts-detection** skill ([../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md)):
- Unnecessary abstractions (interfaces with single impl)
- Factory patterns for simple objects
- Excessive defensive coding
- Over-configuration

### 6. Analyze Dead Code

- Unused imports (grep for imports, check usage)
- TODO/FIXME comments
- Commented-out code blocks
- Unused functions/variables

## Output Format

Return ONLY the JSON object. No markdown, no explanations. The JSON must be valid and follow [fact-schema.md](fact-schema.md).

## Important Rules

1. **Do not score** - Only gather facts
2. **Be thorough** - Check all changed files
3. **Be specific** - Include file:line references
4. **Be objective** - Report what you find, not opinions
5. **Use the skill** - Load the **llm-artifacts-detection** skill ([../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md)) for dead code/overengineering
rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).Stars: 198kUpdated Claude CodeCodexUnverifiedInstall
doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks.Stars: 198kUpdated Claude CodeCodexVerifiedInstall