Skip to content

jhhuh/experiment-driven-skill-development / skills/blind-skill-assessment

blind-skill-assessment

Verified

Use when comparing two versions of agent output to determine which is better, or when evaluating whether a skill produces higher quality results than baseline

Blind Skill Assessment

Core Principle

No baseline, no experiment. Every assessment compares two versions under blinded conditions with structured scoring.

Process

1. BLIND     — Randomly assign labels A/B. Record mapping privately.
               Strip origin hints (filenames, "baseline"/"skill" comments).
2. RUBRIC    — State scoring dimensions before reading the code.
3. JUDGE     — Three personas score both versions (1-5 per dimension).
4. DECODE    — Reveal A/B mapping only after ALL scoring is complete.
5. AGGREGATE — Tally dimension wins, compute per-persona averages.

Default Personas

PersonaFocusDimensions
Bug HunterCorrectnessBugs, edge cases, error handling, race conditions
ArchitectDesignModularity, separation of concerns, extensibility
PragmatistClarityReadability, naming, documentation, maintainability

Each persona scores 1-5 per dimension for both A and B, then picks a per-dimension winner. Append confidence: [h] [m] [l].

Scoring Anchors

  • 1 Broken 2 Significant issues 3 Adequate 4 Good, minor issues 5 Excellent

Anti-Bias Rules

  • Judges never see origin labels ("baseline", "skill-version", "v1", "v2").
  • Randomize which version is A. Do not always assign the first-listed input as A.
  • Score each dimension independently — no holistic verdicts.
  • Sanitize identifying information (file paths, naming conventions that leak origin).

Custom Personas

Add a domain-specific persona when the defaults don't cover the task's concerns. Each needs: name (role title), focus area (one sentence), scoring anchors (what 1 and 5 mean).

Aggregation

  1. Per-experiment: Dimension wins across all personas. More wins takes the experiment.
  2. Cross-experiment: Tally experiment wins as "X/N won by [version]."
  3. Per-persona: Average scores to identify where improvement is strongest/weakest.

Example

Task: Compare two implementations of merge3().

## Label Assignment (private)
Coin flip: tails → Version A = skill-version, Version B = baseline

## Bug Hunter — Correctness
           A    B
Edge cases: 4[h] 3[h]  — A handles empty-file edge case, B does not
Error path: 3[m] 3[m]  — both miss error on binary input
Winner: A

## Architect — Design
              A    B
Modularity:    4[h] 3[h]  — A separates hunk extraction cleanly
API surface:   3[m] 4[m]  — B's top-level API has better early-exit
Winner: tie

## Pragmatist — Clarity
              A    B
Readability:   3[h] 4[h]  — B's variable names are clearer
Documentation: 4[m] 3[m]  — A has better docstring coverage
Winner: tie

## Decode
A = skill-version, B = baseline

## Result
Dimension wins: A=3, B=2, ties=1 → A wins this experiment.

Red Flags — STOP

  • Evaluating without randomizing labels
  • Reading origin labels before scoring is complete
  • Giving only a holistic verdict with no per-dimension scores
  • Skipping the upfront rubric ("I'll just see what stands out")
  • Using a single perspective instead of multiple personas

If you catch yourself doing any of these: STOP. Randomize. Score per-dimension. Then proceed.

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 319kUpdated Claude CodeCodexVerifiedInstall