softaworks/agent-toolkit / skills/skill-judge
skill-judge
VerifiedEvaluate Agent Skill design quality against official specifications and best practices. Use when reviewing, auditing, or improving SKILL.md files and skill packages. Provides multi-dimensional scoring and actionable improvement suggestions.
README.md
View on GitHub# Skill Judge
A comprehensive evaluation framework for assessing Agent Skill quality against official specifications and best practices. This skill provides multi-dimensional scoring and actionable improvement suggestions for SKILL.md files and skill packages.
## Purpose
Skill Judge exists to solve a critical problem: **most Skills waste tokens on knowledge Claude already has**.
The skill helps you evaluate whether a Skill actually adds value by measuring its "knowledge delta" - the gap between what the Skill provides and what Claude already knows. A good Skill should be a compressed expert brain, not a tutorial.
### The Core Formula
> **Good Skill = Expert-only Knowledge - What Claude Already Knows**
This skill helps you identify:
- Token-wasting redundant content (things Claude already knows)
- Genuine expert knowledge that adds value
- Structural issues that prevent Skills from being activated or used effectively
## When to Use
Use Skill Judge when you need to:
- **Review a Skill before publishing**: Evaluate quality and identify improvements
- **Audit existing Skills**: Systematic assessment against best practices
- **Improve a SKILL.md file**: Get specific, actionable suggestions
- **Learn Skill design patterns**: Understand what makes a great Skill
- **Compare Skills**: Assess relative quality using consistent criteria
**Trigger phrases**:
- "Evaluate this skill"
- "Review my SKILL.md"
- "Audit this skill"
- "Score this skill"
- "How can I improve this skill?"
- "Is this skill well-designed?"
## How It Works
### Evaluation Protocol
1. **First Pass - Knowledge Delta Scan**: Read the SKILL.md and categorize each section as:
- **[E] Expert**: Claude genuinely doesn't know this (value-add)
- **[A] Activation**: Claude knows but brief reminder is useful (acceptable)
- **[R] Redundant**: Claude definitely knows this (should delete)
2. **Structure Analysis**: Check frontmatter validity, line count, reference files, design pattern, and loading triggers
3. **Score Each Dimension**: Evaluate against 8 dimensions with specific evidence and justifications
4. **Calculate Total and Grade**: Sum scores (max 120 points) and assign grade
5. **Generate Report**: Produce structured report with scores, critical issues, and improvements
### The 8 Evaluation Dimensions (120 points total)
| Dimension | Max Points | What It Measures |
|-----------|------------|------------------|
| **D1: Knowledge Delta** | 20 | Does the Skill add genuine expert knowledge? (THE CORE DIMENSION) |
| **D2: Mindset + Procedures** | 15 | Does it transfer expert thinking patterns and domain-specific workflows? |
| **D3: Anti-Pattern Quality** | 15 | Does it have effective NEVER lists with specific reasons? |
| **D4: Specification Compliance** | 15 | Is the frontmatter valid? Is the description comprehensive? |
| **D5: Progressive Disclosure** | 15 | Is content properly layered for on-demand loading? |
| **D6: Freedom Calibration** | 15 | Is specificity appropriate for task fragility? |
| **D7: Pattern Recognition** | 10 | Does it follow an established official pattern? |
| **D8: Practical Usability** | 15 | Can an Agent actually use this Skill effectively? |
### Grading Scale
| Grade | Percentage | Meaning |
|-------|------------|---------|
| A | 90%+ (108+) | Excellent - production-ready expert Skill |
| B | 80-89% (96-107) | Good - minor improvements needed |
| C | 70-79% (84-95) | Adequate - clear improvement path |
| D | 60-69% (72-83) | Below Average - significant issues |
| F | <60% (<72) | Poor - needs fundamental redesign |
## Key Features
### Knowledge Classification System
The skill teaches you to recognize three types of content:
| Type | Definition | Treatment |
|------|------------|-----------|
| **Expert** | Claude genuinely doesn't know this | Must keep - this is the Skill's value |
| **Activation** | Claude knows but may not think of | Keep if brief - serves as reminder |
| **Redundant** | Claude definitely knows this | Should delete - wastes tokens |
### Five Official Design Patterns
Skill Judge identifies and evaluates against five established patterns:
| Pattern | Lines | Best For | Example |
|---------|-------|----------|---------|
| **Mindset** | ~50 | Creative tasks requiring taste | frontend-design |
| **Navigation** | ~30 | Multiple distinct scenarios | internal-comms |
| **Philosophy** | ~150 | Art/creation requiring originality | canvas-design |
| **Process** | ~200 | Complex multi-step projects | mcp-builder |
| **Tool** | ~300 | Precise operations on specific formats | docx, pdf, xlsx |
### Common Failure Pattern Detection
The skill identifies 9 common failure patterns:
1. **The Tutorial**: Explains basics Claude already knows
2. **The Dump**: Everything in one 800+ line file
3. **The Orphan References**: Reference files that never get loaded
4. **The Checkbox Procedure**: Mechanical steps without thinking frameworks
5. **The Vague Warning**: "Be careful" without specific guidance
6. **The Invisible Skill**: Great content but poor description
7. **The Wrong Location**: Trigger info in body instead of description
8. **The Over-Engineered**: Unnecessary auxiliary files
9. **The Freedom Mismatch**: Wrong freedom level for task type
## Usage Examples
### Basic Evaluation
```
Evaluate the skill at skills/my-new-skill/SKILL.md
```
### Comparative Analysis
```
Compare the quality of skills/skill-a and skills/skill-b
```
### Targeted Improvement
```
How can I improve the knowledge delta in my skill?
```
### Pattern Identification
```
What pattern does this skill follow, and is it the right choice?
```
## Output
Skill Judge produces a structured evaluation report:
```markdown
# Skill Evaluation Report: [Skill Name]
## Summary
- **Total Score**: X/120 (X%)
- **Grade**: [A/B/C/D/F]
- **Pattern**: [Mindset/Navigation/Philosophy/Process/Tool]
- **Knowledge Ratio**: E:A:R = X:Y:Z
- **Verdict**: [One sentence assessment]
## Dimension Scores
[Table with scores for all 8 dimensions]
## Critical Issues
[Must-fix problems]
## Top 3 Improvements
[Prioritized improvement suggestions]
## Detailed Analysis
[In-depth analysis for dimensions scoring below 80%]
```
## Best Practices
### When Evaluating Skills
**Do:**
- Always check the description field first (it's the most critical)
- Ask "Does Claude already know this?" for every section
- Look for specific anti-patterns with non-obvious reasons
- Verify decision trees actually lead to correct choices
- Check that loading triggers are embedded in workflows
**Never:**
- Give high scores just because content looks professional
- Ignore token waste from redundant explanations
- Let length impress you (43 lines can outperform 500)
- Forgive explaining basics as "helpful context"
- Put "when to use" information only in the body
### The Meta-Question
When evaluating any Skill, always ask:
> "Would an expert in this domain, looking at this Skill, say: 'Yes, this captures knowledge that took me years to learn'?"
If yes, the Skill has genuine value. If no, it's compressing what Claude already knows.
## Quick Reference Checklist
```
SKILL EVALUATION QUICK CHECK
KNOWLEDGE DELTA (most important):
[ ] No "What is X" explanations for basic concepts
[ ] No step-by-step tutorials for standard operations
[ ] Has decision trees for non-obvious choices
[ ] Has trade-offs only experts would know
[ ] Has edge cases from real-world experience
MINDSET + PROCEDURES:
[ ] Transfers thinking patterns (how to think about problems)
[ ] Has "Before doing X, ask yourself..." frameworks
[ ] Includes domain-specific procedures Claude wouldn't know
ANTI-PATTERNS:
[ ] Has explicit NEVER list
[ ] Anti-patterns are specific, not vague
[ ] Includes WHY (non-obvious reasons)
SPECIFICATION:
[ ] Valid YAML frontmatter
[ ] Description answers: WHAT, WHEN, KEYWORDS
[ ] Description specific enough for Agent activation
STRUCTURE:
[ ] SKILL.md < 500 lines (ideal < 300)
[ ] Loading triggers embedded in workflow
[ ] Has "Do NOT Load" for preventing over-loading
FREEDOM:
[ ] Creative tasks -> High freedom (principles)
[ ] Fragile operations -> Low freedom (exact scripts)
USABILITY:
[ ] Decision trees for multi-path scenarios
[ ] Working code examples
[ ] Error handling and fallbacks
```
## Prerequisites
None. Skill Judge is self-contained and requires no external tools or dependencies.
## Related Concepts
- **Tool vs Skill**: Tools define capability boundaries (what Claude CAN do). Skills inject knowledge (what Claude KNOWS how to do).
- **Progressive Disclosure**: Three-layer loading system (metadata -> SKILL.md body -> resources)
- **Freedom Calibration**: Matching constraint level to task fragility
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |