jhhuh/experiment-driven-skill-development / skills/blind-skill-assessment
blind-skill-assessment
VerifiedUse when comparing two versions of agent output to determine which is better, or when evaluating whether a skill produces higher quality results than baseline
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.4 KB |
SKILL.md
View on GitHub---
name: blind-skill-assessment
description: Use when comparing two versions of agent output to determine which is better, or when evaluating whether a skill produces higher quality results than baseline
---
# Blind Skill Assessment
## Core Principle
No baseline, no experiment. Every assessment compares two versions under blinded conditions with structured scoring.
## Process
```
1. BLIND — Randomly assign labels A/B. Record mapping privately.
Strip origin hints (filenames, "baseline"/"skill" comments).
2. RUBRIC — State scoring dimensions before reading the code.
3. JUDGE — Three personas score both versions (1-5 per dimension).
4. DECODE — Reveal A/B mapping only after ALL scoring is complete.
5. AGGREGATE — Tally dimension wins, compute per-persona averages.
```
## Default Personas
| Persona | Focus | Dimensions |
|---------|-------|------------|
| **Bug Hunter** | Correctness | Bugs, edge cases, error handling, race conditions |
| **Architect** | Design | Modularity, separation of concerns, extensibility |
| **Pragmatist** | Clarity | Readability, naming, documentation, maintainability |
Each persona scores 1-5 per dimension for both A and B, then picks a per-dimension winner. Append confidence: `[h]` `[m]` `[l]`.
### Scoring Anchors
- **1** Broken **2** Significant issues **3** Adequate **4** Good, minor issues **5** Excellent
## Anti-Bias Rules
- Judges never see origin labels ("baseline", "skill-version", "v1", "v2").
- Randomize which version is A. Do not always assign the first-listed input as A.
- Score each dimension independently — no holistic verdicts.
- Sanitize identifying information (file paths, naming conventions that leak origin).
## Custom Personas
Add a domain-specific persona when the defaults don't cover the task's concerns. Each needs: **name** (role title), **focus area** (one sentence), **scoring anchors** (what 1 and 5 mean).
## Aggregation
1. **Per-experiment:** Dimension wins across all personas. More wins takes the experiment.
2. **Cross-experiment:** Tally experiment wins as "X/N won by [version]."
3. **Per-persona:** Average scores to identify where improvement is strongest/weakest.
## Example
Task: Compare two implementations of `merge3()`.
```
## Label Assignment (private)
Coin flip: tails → Version A = skill-version, Version B = baseline
## Bug Hunter — Correctness
A B
Edge cases: 4[h] 3[h] — A handles empty-file edge case, B does not
Error path: 3[m] 3[m] — both miss error on binary input
Winner: A
## Architect — Design
A B
Modularity: 4[h] 3[h] — A separates hunk extraction cleanly
API surface: 3[m] 4[m] — B's top-level API has better early-exit
Winner: tie
## Pragmatist — Clarity
A B
Readability: 3[h] 4[h] — B's variable names are clearer
Documentation: 4[m] 3[m] — A has better docstring coverage
Winner: tie
## Decode
A = skill-version, B = baseline
## Result
Dimension wins: A=3, B=2, ties=1 → A wins this experiment.
```
## Red Flags — STOP
- Evaluating without randomizing labels
- Reading origin labels before scoring is complete
- Giving only a holistic verdict with no per-dimension scores
- Skipping the upfront rubric ("I'll just see what stands out")
- Using a single perspective instead of multiple personas
**If you catch yourself doing any of these: STOP. Randomize. Score per-dimension. Then proceed.**
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |