borghei/Claude-Skills / engineering/skill-tester
skill-tester
VerifiedValidate and score Claude Code skill packages for quality, completeness, and best-practice compliance. Tests Python scripts, checks YAML frontmatter, and generates reports. Use when creating, validating, or auditing skill packages.
Skill Tester
Validate skill packages for structure compliance, test Python scripts for syntax and stdlib-only imports, and score quality across four dimensions (documentation, code quality, completeness, usability) with letter grades and improvement recommendations. Supports BASIC, STANDARD, and POWERFUL tier classification.
Core Capabilities
- Structure validation — check required files, directory layout, YAML frontmatter, and required SKILL.md sections against tier thresholds.
- Script testing — AST syntax checks, stdlib-only import analysis, argparse and
__main__-guard detection, runtime--helpand sample-data execution. - Quality scoring — four equally weighted dimensions producing an overall score, letter grade (A+ through F), and a prioritized improvement roadmap.
- Tier classification — BASIC / STANDARD / POWERFUL requirements for SKILL.md depth, script count/LOC, argparse, output formats, and error handling.
- Dual output — human-readable reports and
--jsonfor CI/CD gating with meaningful exit codes.
When to Use
- Creating a new skill and validating it before publishing.
- Auditing an existing skill's structure, scripts, and quality.
- Embedding a quality gate into a CI/CD pipeline.
Clarify First
Before validating, settle the inputs below. When one is missing or vague and would change the result, ask about it; when it wouldn't, use a sensible default and say which one you used.
- Target skill path: the skill directory to validate, test, and score (the subject of all three tools).
- Target tier: BASIC / STANDARD / POWERFUL (
--tier; sets the required sections, script count, and structural thresholds). - Pass bar: the minimum quality score / whether failures gate CI (
--minimum-score, exit codes).
Ask about the two or three inputs that matter most for this request, not the whole list. If the user wants a result straight away, produce it and state your assumptions at the top.
Quick Start
# Validate the bundled fixture skill, then score it
python scripts/skill_validator.py assets/sample-skill --tier BASIC
python scripts/quality_scorer.py assets/sample-skill --detailed --minimum-score 75
# Run its scripts (this EXECUTES the target's code; see the warning below)
python scripts/script_tester.py assets/sample-skill --timeout 30
Warning: script_tester.py runs the target skill's scripts (--help and their sample
inputs) as you. Never point it at a skill you have not read. For a third-party skill, run
skill-security-auditor first, and run the tester in a disposable sandbox without
credentials or network.
Tools
| Tool | Purpose | Command |
|---|---|---|
skill_validator.py | Validate structure, frontmatter, required sections, and scripts against tier rules | python scripts/skill_validator.py engineering/my-skill --tier POWERFUL --json |
script_tester.py | Static + runtime tests of scripts (syntax, imports, argparse, --help, samples) | python scripts/script_tester.py engineering/my-skill --timeout 60 --json |
quality_scorer.py | Score four quality dimensions with letter grade and improvement roadmap | python scripts/quality_scorer.py engineering/my-skill --detailed --minimum-score 75 --json |
Exit codes: skill_validator.py 0 pass, 1 errors or score below 60, 2 skill path not found;
script_tester.py 0 all pass, 1 failure (including a missing path), 2 partial pass;
quality_scorer.py 0 grades A-C, 1 grade F, below --minimum-score or missing path,
2 grade D. The length checks are bands, not minimums: SKILL.md passes at 40-500 non-blank
lines and scores best at 60-500; scripts score best at 100-300 lines, because a lean
SKILL.md with detail in references/ is the goal.
See references/tool-reference.md for full parameter tables and output formats.
References
Load the reference that matches the task — keep this file lean and pull detail on demand:
- references/workflows-and-cicd.md — the three core validation workflows, the tier-requirements and quality-scoring tables, CI/CD integration, anti-patterns, troubleshooting, and success criteria. Read when running a validation pass or wiring a CI gate.
- references/tool-reference.md — full parameter tables, output formats, and exit codes for the three Python tools. Read when scripting the tools or interpreting JSON output.
- references/skill-structure-specification.md — the authoritative specification for skill directory structure, required files, and frontmatter. Read when defining what "valid structure" means.
- references/tier-requirements-matrix.md — the full BASIC/STANDARD/POWERFUL requirements matrix with detailed criteria per tier. Read when classifying or upgrading a skill's tier.
- references/quality-scoring-rubric.md — the detailed scoring rubric with per-component weights and grading bands. Read when interpreting or tuning quality scores.
Scope & Limitations
Covers:
- Structural validation of skill directories against tier-specific requirements (BASIC, STANDARD, POWERFUL)
- Static analysis of Python scripts including syntax checking, import validation, argparse detection, and main guard verification
- Multi-dimensional quality scoring across documentation, code quality, completeness, and usability
- Dual output formatting (JSON for CI/CD pipelines, human-readable for developer consumption)
Does NOT cover:
- Functional correctness of script logic or algorithm accuracy — the tester verifies structure and conventions, not business logic
- Performance benchmarking or memory profiling of scripts — see
engineering/performance-profilerfor runtime analysis - Security vulnerability scanning of script code — see
engineering/skill-security-auditorfor dependency and code security audits - Cross-skill dependency resolution or integration testing — skills are validated in isolation without verifying inter-skill compatibility
Integration Points
| Skill | Integration | Data Flow |
|---|---|---|
engineering/skill-security-auditor | Run security audit after validation passes | skill_validator.py confirms structure compliance, then skill-security-auditor scans for vulnerabilities in the same skill path |
engineering/ci-cd-pipeline-builder | Embed skill-tester as a quality gate stage | Pipeline builder generates workflow YAML that invokes skill_validator.py, script_tester.py, and quality_scorer.py sequentially |
engineering/changelog-generator | Feed quality score deltas into changelog entries | Compare quality_scorer.py JSON output between releases to surface quality improvements or regressions |
engineering/pr-review-expert | Attach validation report to pull request reviews | skill_validator.py --json output is posted as a PR comment for reviewer context |
engineering/performance-profiler | Complement structural testing with runtime profiling | After script_tester.py confirms execution succeeds, performance-profiler measures execution time and resource usage |
engineering/tech-debt-tracker | Track quality score trends over time | Periodic quality_scorer.py --json output is ingested to detect score degradation and flag technical debt |
Anti-Patterns
Padding a Skill to Pass a Length Check
Mistake: Adding filler sections so SKILL.md looks "comprehensive".
Why it happens: Older scorers rewarded line count.
Instead: Keep SKILL.md to goals, decision points, tools and verification; move reference tables to references/. The validator now checks a band, not a minimum.
Testing an Untrusted Skill by Running It
Mistake: Running script_tester.py on a downloaded skill to see whether it is safe.
Why it happens: "Test" sounds harmless.
Instead: The tester executes the target's code. Audit it statically first (skill-security-auditor), then test in a sandbox.
Gating Only on the Grade
Mistake: Merging any skill that scores B or better.
Why it happens: A single grade is easy to wire into CI.
Instead: Also require script_tester.py to pass and read the validator's warnings; a well-formatted skill can still ship a broken tool.
Files
| File | Purpose |
|---|---|
scripts/skill_validator.py | Structure, frontmatter, section and script checks per tier |
scripts/script_tester.py | Static and runtime checks of a skill's scripts (executes them) |
scripts/quality_scorer.py | Four-dimension quality score, grade and roadmap |
assets/sample-skill/ | Fixture skill for trying the tools |
expected_outputs/ | Reference outputs for the fixture |
references/*.md | Tool reference, structure spec, tier matrix, scoring rubric, CI workflows |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |