Skip to content

The-AI-Directory-Company/agents-and-skills / skills/hiring-rubric

hiring-rubric

Verified

Build structured hiring rubrics — defining evaluation dimensions, behavioral interview questions, scoring criteria, and calibration processes that maximize signal and minimize bias.

references/calibration-workshop-guide.md

View on GitHub
# Calibration Workshop Guide

How to run a pre-interview calibration session that aligns the interview panel on scoring standards before any candidate walks through the door.

## Why Calibrate

Without calibration, a "3" means something different to every interviewer. One person's "Hire" is another's "Strong Hire." The debrief becomes a negotiation about standards instead of a discussion about the candidate. Calibration fixes this by anchoring the panel to shared examples before interviews begin.

## When to Run

- **Before every new role's interview loop begins.** Not once per quarter — once per role.
- **When the panel changes.** A new interviewer joining mid-loop needs calibration, even if others already completed it.
- **Duration:** 60-90 minutes. Do not try to compress this into 30 minutes.

## Pre-Workshop Preparation

### 1. Select Mock Clips (Owner: Hiring Manager)

Choose 2-3 recorded interview clips or written transcripts that represent different performance levels. Ideal sources:

- Past interview recordings (anonymized) at different score levels
- Role-played interviews recorded by team members
- Written vignettes if recordings are unavailable

**Clip selection criteria:**

| Clip | Target Score | Why This Clip |
|------|-------------|---------------|
| Clip A | 2 (Weak) | Shows a common failure mode for this role — plausible but flawed |
| Clip B | 3 (Hire) | Solid performance with minor gaps — the hardest score to calibrate |
| Clip C | 4 (Strong Hire) | Clearly exceptional, but the specific "why" is worth discussing |

The 3 is the most important clip. Most disagreements happen at the Hire/Weak boundary.

### 2. Distribute Materials (2 days before)

Send to all panelists:
- The role's hiring rubric (dimensions, weights, scoring criteria)
- Blank scorecards
- The mock clips or transcripts
- Instructions: "Score each clip independently before the workshop. Bring your completed scorecards."

## Workshop Agenda

### Part 1: Independent Scoring Review (15 minutes)

Confirm everyone has scored the clips. Do not allow scoring during the meeting — it defeats the purpose. If someone did not complete the pre-work, they observe but do not participate in scoring discussions.

### Part 2: Score Reveal — Clip by Clip (30-40 minutes)

For each clip:

1. **Silent reveal.** Everyone holds up their score simultaneously (use a shared doc, chat, or literal fingers). No one speaks first.
2. **Identify disagreements.** Note where scores differ by 2+ points. Minor variance (3 vs. 4) is normal; large gaps (2 vs. 4) indicate a calibration problem.
3. **Lowest scorer speaks first.** The person who gave the lowest score explains their reasoning with specific evidence from the clip. This prevents anchoring to the majority view.
4. **Highest scorer responds.** They explain what they saw differently, citing specific moments.
5. **Group discussion.** Focus on: "What specific behavior would move this from a 2 to a 3?" or "What was missing that kept this from being a 4?"
6. **Agree on the reference score.** The group does not need to reach unanimous agreement, but the hiring manager should document the consensus score and the key behavioral markers that justify it.

### Part 3: Anchor Language (15-20 minutes)

For each dimension, write down 1-2 sentences that complete these phrases:

- "A 3 on [dimension] means the candidate..."
- "The difference between a 2 and a 3 on [dimension] is..."
- "A 4 requires the candidate to... without prompting."

Write these in shared notes. Interviewers should reference them when filling out scorecards during actual interviews.

### Part 4: Process Alignment (10 minutes)

Confirm logistics:
- Scorecard submission deadline (within 24 hours of interview)
- Debrief format (most junior interviewer presents first)
- Escalation path for split decisions
- Reminder: scores must be submitted before debrief — no post-hoc adjustments

## Disagreement Resolution Patterns

When the panel cannot agree on a reference score, use these techniques:

### Behavioral Decomposition

Break the disagreement into specific behaviors: "We agree they identified the edge case (that's a 3-level behavior for problem-solving). The question is whether proposing the fix unprompted makes it a 4 or whether the fix itself was too shallow to qualify."

### Threshold Questions

Ask: "If this were the only data point we had for this dimension, would you hire this person?" If the answer splits, the clip is genuinely borderline — document it as a 2-3 boundary example rather than forcing consensus.

### Role-Specific Calibration

For dimensions that are unusually important to this role, tighten the standard: "For a senior infrastructure engineer, a 3 on system design requires multi-region considerations. For a junior role, single-region with acknowledged limitations is a 3."

## Anti-Patterns to Avoid

- **Calibrating on perfect candidates.** Use clips that show flawed-but-reasonable performance. A clip where everything goes right does not help distinguish 3 from 4.
- **Hiring manager anchoring.** The hiring manager should reveal their score last, not first. If they speak first, the room adjusts to them.
- **Skipping the 3.** Everyone can identify a 1 and a 4. The 3 is where calibration matters most. Spend the majority of your time on Hire-level clips.
- **One-time calibration for ongoing roles.** If you run the same role across multiple quarters, re-calibrate when the panel changes or when debrief disagreements become frequent.
- **Rushing the workshop.** Cutting it to 30 minutes means you skip the anchor language step, which is where the lasting value is. Protect the full 60-90 minutes.

## Output Artifacts

By the end of the workshop, the hiring manager should have:

1. **Reference scores** for 2-3 clips with documented reasoning
2. **Anchor language** for each dimension at the 2/3 and 3/4 boundaries
3. **Process confirmation** — submission deadlines, debrief order, escalation path
4. **Attendance record** — who calibrated, who still needs to before interviewing
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 196kUpdated Claude CodeCodexVerifiedInstall