Skip to content

borghei/Claude-Skills / engineering/prompt-engineer-toolkit

prompt-engineer-toolkit

Verified

Tests, versions and scores production prompts: prompt linting, version diffs, eval-suite scoring with a CI gate. Use when changing a production prompt, building an eval suite, debugging a regression, or migrating to a new model.

FileKindSize
SKILL.mdskill md7.8 KB
assets/prompt_v1.txtmarkdown241 B
assets/prompt_v2.txtmarkdown973 B
assets/sample_eval_suite.jsonconfig996 B
references/failure-modes-and-workflows.mdmarkdown5.6 KB
references/prompt-patterns-catalog.mdmarkdown8.5 KB
references/testing-and-versioning.mdmarkdown3.8 KB
scripts/eval_scorer.pyexecutablescript14.4 KB
scripts/prompt_analyzer.pyexecutablescript10.3 KB
scripts/prompt_diff.pyexecutablescript11.9 KB

references/testing-and-versioning.md

View on GitHub
# Prompt Testing & Versioning

Read this when building a test suite, scoring prompt quality, or managing prompt versions — the testing framework, evaluation rubric, regression protocol, version-control strategy, changelog format, and diff analysis.

## Prompt Testing Framework

### Test Case Design

Every production prompt needs a test suite.

#### Test Case Structure

```json
{
  "test_id": "classify-urgent-001",
  "input": "Server is down, customers can't access the product",
  "expected": {
    "contains": ["critical", "immediate"],
    "not_contains": ["low priority", "can wait"],
    "format_regex": "^\\{.*\\}$",
    "max_tokens": 500,
    "required_fields": ["severity", "category"]
  },
  "tags": ["classification", "urgency", "happy-path"]
}
```

#### Test Suite Composition

| Category | % of Suite | Purpose |
|----------|-----------|---------|
| Happy path | 40% | Confirm basic functionality works |
| Edge cases | 30% | Boundary conditions, unusual inputs |
| Adversarial | 15% | Inputs designed to break the prompt |
| Regression | 15% | Cases that previously failed |

### Evaluation Rubric

#### Automated Scoring

| Dimension | Measurement | Weight |
|-----------|-------------|--------|
| Adherence | Contains required elements, matches schema | 30% |
| Accuracy | Correct classification/analysis/answer | 30% |
| Safety | No forbidden content, no hallucinations | 20% |
| Format | Matches expected structure, length bounds | 10% |
| Relevance | Response addresses the actual input | 10% |

#### Scoring Formula

```
score = (adherence * 0.30) + (accuracy * 0.30) + (safety * 0.20) + (format * 0.10) + (relevance * 0.10)

Pass threshold: 0.80
Warning threshold: 0.70
Fail threshold: < 0.70
```

### Regression Testing Protocol

```
1. Before any prompt change:
   - Run full test suite against current prompt (baseline)
   - Record scores per test case

2. After prompt change:
   - Run same test suite against new prompt (candidate)
   - Compare scores per test case

3. Acceptance criteria:
   - Average score: candidate >= baseline
   - No individual test case drops by more than 10%
   - Zero safety violations (any safety failure = reject)
   - If criteria met: promote candidate
   - If criteria not met: iterate on prompt or reject
```

## Prompt Versioning

### Version Control Strategy

```
prompts/
├── support-classifier/
│   ├── v1.txt                 # Original version
│   ├── v2.txt                 # Added edge case handling
│   ├── v3.txt                 # Current production
│   ├── changelog.md           # Change log with rationale
│   └── tests/
│       ├── suite.json         # Test cases
│       └── baselines/
│           ├── v1-results.json
│           ├── v2-results.json
│           └── v3-results.json
├── code-reviewer/
│   ├── v1.txt
│   └── ...
```

### Changelog Format

```markdown
## v3 (2026-03-09)
**Author:** borghei
**Change:** Added explicit handling for multi-language inputs
**Reason:** v2 defaulted to English analysis for non-English code comments
**Test results:** Average score 0.87 (v2 was 0.82). No regressions.
**Rollback plan:** Revert to v2.txt

## v2 (2026-02-15)
**Author:** borghei
**Change:** Added structured output format with JSON schema
**Reason:** Downstream parser needed consistent format
**Test results:** Average score 0.82 (v1 was 0.79). Format compliance 100% (v1 was 73%).
```

### Prompt Diff Analysis

Before deploying a new version, always diff:

```
Key questions for prompt diffs:
1. Were any constraints removed? (Risk: safety regression)
2. Were any examples changed? (Risk: calibration shift)
3. Was the output format changed? (Risk: downstream parser breaks)
4. Were any anti-patterns removed? (Risk: known failure modes return)
5. Is the new prompt longer? (Risk: context budget impact)
```
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall