Skip to content

borghei/Claude-Skills / engineering/skill-tester

skill-tester

Verified

Validate and score Claude Code skill packages for quality, completeness, and best-practice compliance. Tests Python scripts, checks YAML frontmatter, and generates reports. Use when creating, validating, or auditing skill packages.

---
name: skill-tester
description: >
  Validate and score Claude Code skill packages for quality, completeness, and best-practice
  compliance. Tests Python scripts, checks YAML frontmatter, and generates reports. Use when
  creating, validating, or auditing skill packages.
license: MIT + Commons Clause
metadata:
  version: 1.1.0
  author: borghei
  category: engineering
  domain: meta-skills
  tier: POWERFUL
  updated: 2026-11-08
  tags:
    - skill-validation
    - quality-scoring
    - frontmatter
    - testing
    - meta-skills
---
# Skill Tester

Validate skill packages for structure compliance, test Python scripts for syntax and stdlib-only imports, and score quality across four dimensions (documentation, code quality, completeness, usability) with letter grades and improvement recommendations. Supports BASIC, STANDARD, and POWERFUL tier classification.

## Core Capabilities

- **Structure validation** — check required files, directory layout, YAML frontmatter, and required SKILL.md sections against tier thresholds.
- **Script testing** — AST syntax checks, stdlib-only import analysis, argparse and `__main__`-guard detection, runtime `--help` and sample-data execution.
- **Quality scoring** — four equally weighted dimensions producing an overall score, letter grade (A+ through F), and a prioritized improvement roadmap.
- **Tier classification** — BASIC / STANDARD / POWERFUL requirements for SKILL.md depth, script count/LOC, argparse, output formats, and error handling.
- **Dual output** — human-readable reports and `--json` for CI/CD gating with meaningful exit codes.

## When to Use

- Creating a new skill and validating it before publishing.
- Auditing an existing skill's structure, scripts, and quality.
- Embedding a quality gate into a CI/CD pipeline.

## Clarify First

Before validating, settle the inputs below. When one is missing or vague and would change the result, ask about it; when it wouldn't, use a sensible default and say which one you used.

- **Target skill path:** the skill directory to validate, test, and score (the subject of all three tools).
- **Target tier:** BASIC / STANDARD / POWERFUL (`--tier`; sets the required sections, script count, and structural thresholds).
- **Pass bar:** the minimum quality score / whether failures gate CI (`--minimum-score`, exit codes).

Ask about the two or three inputs that matter most for this request, not the whole list. If the user wants a result straight away, produce it and state your assumptions at the top.

## Quick Start

```bash
# Validate the bundled fixture skill, then score it
python scripts/skill_validator.py assets/sample-skill --tier BASIC
python scripts/quality_scorer.py assets/sample-skill --detailed --minimum-score 75

# Run its scripts (this EXECUTES the target's code; see the warning below)
python scripts/script_tester.py assets/sample-skill --timeout 30
```

**Warning: `script_tester.py` runs the target skill's scripts** (`--help` and their sample
inputs) as you. Never point it at a skill you have not read. For a third-party skill, run
`skill-security-auditor` first, and run the tester in a disposable sandbox without
credentials or network.

## Tools

| Tool | Purpose | Command |
|------|---------|---------|
| `skill_validator.py` | Validate structure, frontmatter, required sections, and scripts against tier rules | `python scripts/skill_validator.py engineering/my-skill --tier POWERFUL --json` |
| `script_tester.py` | Static + runtime tests of scripts (syntax, imports, argparse, `--help`, samples) | `python scripts/script_tester.py engineering/my-skill --timeout 60 --json` |
| `quality_scorer.py` | Score four quality dimensions with letter grade and improvement roadmap | `python scripts/quality_scorer.py engineering/my-skill --detailed --minimum-score 75 --json` |

Exit codes: `skill_validator.py` 0 pass, 1 errors or score below 60, 2 skill path not found;
`script_tester.py` 0 all pass, 1 failure (including a missing path), 2 partial pass;
`quality_scorer.py` 0 grades A-C, 1 grade F, below `--minimum-score` or missing path,
2 grade D. The length checks are bands, not minimums: SKILL.md passes at 40-500 non-blank
lines and scores best at 60-500; scripts score best at 100-300 lines, because a lean
SKILL.md with detail in `references/` is the goal.

See **[references/tool-reference.md](references/tool-reference.md)** for full parameter tables and output formats.

## References

Load the reference that matches the task — keep this file lean and pull detail on demand:

- **[references/workflows-and-cicd.md](references/workflows-and-cicd.md)** — the three core validation workflows, the tier-requirements and quality-scoring tables, CI/CD integration, anti-patterns, troubleshooting, and success criteria. Read when running a validation pass or wiring a CI gate.
- **[references/tool-reference.md](references/tool-reference.md)** — full parameter tables, output formats, and exit codes for the three Python tools. Read when scripting the tools or interpreting JSON output.
- **[references/skill-structure-specification.md](references/skill-structure-specification.md)** — the authoritative specification for skill directory structure, required files, and frontmatter. Read when defining what "valid structure" means.
- **[references/tier-requirements-matrix.md](references/tier-requirements-matrix.md)** — the full BASIC/STANDARD/POWERFUL requirements matrix with detailed criteria per tier. Read when classifying or upgrading a skill's tier.
- **[references/quality-scoring-rubric.md](references/quality-scoring-rubric.md)** — the detailed scoring rubric with per-component weights and grading bands. Read when interpreting or tuning quality scores.

## Scope & Limitations

**Covers:**
- Structural validation of skill directories against tier-specific requirements (BASIC, STANDARD, POWERFUL)
- Static analysis of Python scripts including syntax checking, import validation, argparse detection, and main guard verification
- Multi-dimensional quality scoring across documentation, code quality, completeness, and usability
- Dual output formatting (JSON for CI/CD pipelines, human-readable for developer consumption)

**Does NOT cover:**
- Functional correctness of script logic or algorithm accuracy — the tester verifies structure and conventions, not business logic
- Performance benchmarking or memory profiling of scripts — see `engineering/performance-profiler` for runtime analysis
- Security vulnerability scanning of script code — see `engineering/skill-security-auditor` for dependency and code security audits
- Cross-skill dependency resolution or integration testing — skills are validated in isolation without verifying inter-skill compatibility

## Integration Points

| Skill | Integration | Data Flow |
|-------|-------------|-----------|
| `engineering/skill-security-auditor` | Run security audit after validation passes | `skill_validator.py` confirms structure compliance, then `skill-security-auditor` scans for vulnerabilities in the same skill path |
| `engineering/ci-cd-pipeline-builder` | Embed skill-tester as a quality gate stage | Pipeline builder generates workflow YAML that invokes `skill_validator.py`, `script_tester.py`, and `quality_scorer.py` sequentially |
| `engineering/changelog-generator` | Feed quality score deltas into changelog entries | Compare `quality_scorer.py` JSON output between releases to surface quality improvements or regressions |
| `engineering/pr-review-expert` | Attach validation report to pull request reviews | `skill_validator.py --json` output is posted as a PR comment for reviewer context |
| `engineering/performance-profiler` | Complement structural testing with runtime profiling | After `script_tester.py` confirms execution succeeds, `performance-profiler` measures execution time and resource usage |
| `engineering/tech-debt-tracker` | Track quality score trends over time | Periodic `quality_scorer.py --json` output is ingested to detect score degradation and flag technical debt |

## Anti-Patterns

### Padding a Skill to Pass a Length Check
**Mistake:** Adding filler sections so SKILL.md looks "comprehensive".
**Why it happens:** Older scorers rewarded line count.
**Instead:** Keep SKILL.md to goals, decision points, tools and verification; move reference tables to `references/`. The validator now checks a band, not a minimum.

### Testing an Untrusted Skill by Running It
**Mistake:** Running `script_tester.py` on a downloaded skill to see whether it is safe.
**Why it happens:** "Test" sounds harmless.
**Instead:** The tester executes the target's code. Audit it statically first (`skill-security-auditor`), then test in a sandbox.

### Gating Only on the Grade
**Mistake:** Merging any skill that scores B or better.
**Why it happens:** A single grade is easy to wire into CI.
**Instead:** Also require `script_tester.py` to pass and read the validator's warnings; a well-formatted skill can still ship a broken tool.

## Files

| File | Purpose |
|------|---------|
| `scripts/skill_validator.py` | Structure, frontmatter, section and script checks per tier |
| `scripts/script_tester.py` | Static and runtime checks of a skill's scripts (executes them) |
| `scripts/quality_scorer.py` | Four-dimension quality score, grade and roadmap |
| `assets/sample-skill/` | Fixture skill for trying the tools |
| `expected_outputs/` | Reference outputs for the fixture |
| `references/*.md` | Tool reference, structure spec, tier matrix, scoring rubric, CI workflows |
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall