Skip to content

aiskillstore/marketplace / skills/92bilal26/quiz-generator

quiz-generator

Verified

Generate 50-question interactive quizzes using the Quiz component with randomized batching. Use when creating end-of-chapter assessments. Displays 15-20 questions per session with immediate feedback. NOT for static markdown quizzes.

references/option-length-validation.md

View on GitHub
# Option Length Validation Reference

**Purpose:** Ensure all quiz options are balanced in length to prevent test-takers from guessing by selecting longest/shortest options without reading questions.

**Target:** ALL options within ±3 words of each other for EVERY question (all 50 questions)

---

## The Problem

### Why Option Length Matters

Unequal option lengths allow test-takers to achieve 60-70%+ accuracy by pattern-matching:

**Example 1: Length-Based Strategy**
```
Question: "What is mutability?"
A: "Yes"                                                    (2 words)  ← Shortest
B: "It means objects can't be changed after creation"      (9 words)  ← Longest (CORRECT)
C: "No"                                                     (2 words)  ← Shortest
D: "Maybe"                                                  (5 words)  ← Medium
```

**Strategy:** Always pick the longest option
- Expected accuracy: 25% (random)
- Actual accuracy: 75%+ (pattern-matching)
- **Result:** Test measures reading strategy, not understanding

**Example 2: Longest Option Bias**
```
50-question quiz results:
- Longest option is correct in 35 questions (70%)
- Shortest option is correct in 3 questions (6%)
- Middle-length correct in 12 questions (24%)
```

**Pattern:** Students can select longest → 70% accuracy without reading
- **Result:** Quiz is invalid; scores don't reflect understanding

---

## The Solution: ±3 Word Rule

### Standard

**ALL options within ±3 words of each other**

Example acceptable ranges:
- 4, 5, 6, 7 words → PASS (spread: 3 words)
- 10, 11, 12, 13 words → PASS (spread: 3 words)
- 5, 6, 8, 9 words → FAIL (spread: 4 words) ✗ Exceeds ±3 tolerance
- 3, 5, 9, 10 words → FAIL (spread: 7 words)
- 2, 6, 8, 12 words → FAIL (spread: 10 words)

### Why ±3?

**4-5 words:** Similar cognitive load, hard to distinguish by scanning
- "It improves performance" (3 words)
- "It prevents common bugs" (4 words)
- "It simplifies structures" (3 words)

**8+ word difference:** Readers start pattern-matching
- "Yes" (2 words) vs. "The system processes asynchronously" (5 words) → Obvious pattern
- Very short vs. very long → Cognitive load disparities

**±3 sweet spot:**
- Prevents obvious pattern recognition
- Maintains roughly equal cognitive burden
- Prevents length-based guessing strategies

---

## Validation Procedure (Step-by-Step)

### Phase 1: Count Words for All 50 Questions

Create a validation spreadsheet:

```
Q#  | Correct | A (words) | B (words) | C (words) | D (words) | Min-Max | Pass? | Notes
----|---------|-----------|-----------|-----------|-----------|---------|-------|-------
1   | B       | 4         | 5         | 4         | 5         | 4-5     | ✓     | "amplifies practices", etc
2   | A       | 6         | 3         | 7         | 5         | 3-7     | ✗     | Spread=4, FAIL (option B too short)
3   | D       | 4         | 4         | 4         | 4         | 4-4     | ✓     | Perfect parity
...
50  | C       | 5         | 5         | 6         | 5         | 5-6     | ✓     | "AI changes roles", etc
```

### Phase 2: Identify Failing Questions

Flag any question where min-max spread > 3 words:

❌ **FAIL Example:**
```
Question 2: "Why focus on organizational capabilities?"
A: "Developers don't need training if systems are strong" (9 words)
B: "Organizational capabilities create guardrails that prevent failures regardless of developer skill" (11 words)
C: "Tool training is too expensive compared to improvements" (8 words)
D: "AI tools only work well with perfect processes" (8 words)

Spread: 8 to 11 words = 3-word difference ✓ (actually PASS, my mistake in earlier example!)
```

Wait, let me recalculate that example from the user's feedback:

```
A: "Developers don't need training if organizational systems are strong enough" (11 words)
B: "Organizational capabilities (testing, review, deployments) create guardrails that prevent failures regardless of individual developer skill" (14 words)
C: "Tool training is too expensive compared to organizational improvements" (9 words)
D: "AI tools only work well in organizations with perfect processes" (10 words)

Min: 9 words (C)
Max: 14 words (B)
Spread: 14 - 9 = 5 words ✗ FAIL (exceeds ±3 rule)
```

✅ **PASS Example:**
```
Question 5: "What is the purpose of testing?"
A: "It catches breaking changes early" (5 words)
B: "It validates the specification" (4 words)
C: "It ensures code quality" (4 words)
D: "It prevents hidden bugs" (4 words)

Spread: 4 to 5 words = 1-word difference ✓ PASS
```

### Phase 3: Rewrite Failing Questions

For any question failing the ±3 rule, rewrite options to match lengths:

**Before (FAIL):**
```
A: "Yes" (2 words)
B: "Organizational capabilities create guardrails preventing failures" (6 words)
C: "No" (2 words)
D: "Possibly" (5 words)
Spread: 2-6 = 4 words ✗ FAIL
```

**After (PASS):**
```
A: "Systems lack discipline entirely" (4 words)
B: "Organizations create guardrails for safety" (5 words)
C: "Guardrails prevent all problems quickly" (5 words)
D: "Training eliminates organizational weaknesses" (4 words)
Spread: 4-5 = 1 word ✓ PASS
```

### Phase 4: Verify No Length-Correctness Correlation

Create a distribution analysis:

```
Correct Answer Distribution by Length:

Word Count | # of Correct Answers | % of Total
-----------|---------------------|----------
3-4 words  | 8                   | 16%
5-6 words  | 13                  | 26% ← Correct (balanced)
7-8 words  | 12                  | 24%
9+ words   | 17                  | 34% ← BIASED (too many correct)

Result: FAIL - Longest options are disproportionately correct
Fix: Redistribute which options are marked as correct
```

Ideal distribution:
- ~25% correct for shortest options
- ~25% correct for longest options
- ~50% correct for middle-length options

(Why middle? Because most options will be 5-7 words; fewer will be 3-4 or 9+)

### Phase 5: Document Validation Results

Before submitting quiz, include validation report:

```
OPTION LENGTH VALIDATION REPORT
================================

Total Questions: 50
Date Validated: 2025-11-06

RESULTS:
✓ All 50 questions word-counted
✓ All options within ±3 word range
✓ No questions failed length check
✓ Length-correctness correlation verified: NO BIAS DETECTED

DISTRIBUTION ANALYSIS:
- Shortest option correct in 12 questions (24%)
- Longest option correct in 13 questions (26%)
- Middle-length correct in 25 questions (50%)

CONCLUSION: Quiz passes option length validation. Test-takers cannot achieve 60%+ accuracy by selecting longest/shortest option.
```

---

## Practical Tools & Scripts

### Word Count Script (Manual Method)

For each question, count words using this formula:

```
Text: "Organizational capabilities create guardrails that prevent failures"
Count: Organizational(1) capabilities(2) create(3) guardrails(4) that(5) prevent(6) failures(7)
Result: 7 words
```

**Common mistakes:**
- ❌ Don't count punctuation as words: "(testing, review, deployments)" = 3 words, NOT 5
- ❌ Don't count parentheses: "(DORA)" = 0 words, just DORA = 1 word
- ✅ Do count hyphenated words as one: "state-of-the-art" = 1 word
- ✅ Do count contractions as one: "doesn't" = 1 word, "you're" = 1 word

### Spreadsheet Validation (Excel/Google Sheets)

```
Column A: Question Number
Column B: Correct Answer (A/B/C/D)
Column C: Word count for Option A
Column D: Word count for Option B
Column E: Word count for Option C
Column F: Word count for Option D
Column G: =MAX(C2:F2)-MIN(C2:F2)  [calculates spread]
Column H: =IF(G2<=3,"PASS","FAIL")
```

Filter for all FAIL entries → Rewrite those questions

---

## Examples from Real Quiz Questions

### Example 1: PASS (Good Parity)

**Original Question 5:**
```
"According to the DORA research, organizations with weak practices see negative returns from AI adoption.
Why does AI amplify existing problems rather than fix them?"

A: "AI models aren't intelligent enough to compensate for organizational weakness" (10 words)
B: "AI generates code faster, which amplifies the velocity of unvalidated code production and failures if guardrails are weak" (18 words) ❌ TOO LONG
C: "AI tools refuse to work with organizations that lack discipline" (10 words)
D: "Weak practices naturally improve over time; AI accelerates improvement" (9 words)

Spread: 9-18 = 9 words ✗ FAIL
```

**Revised Version (PASS):**
```
A: "AI models lack enough intelligence for weak organizations" (8 words)
B: "AI speeds up code production without organizational guardrails" (8 words) ✓ CORRECT
C: "AI tools automatically enforce discipline in weak teams" (8 words)
D: "Weak practices improve naturally; AI just accelerates them" (8 words)

Spread: 8-8 = 0 words ✓ PASS (Perfect parity)
```

### Example 2: PASS (Within ±3)

**Question 12:**
```
"Why does the chapter argue that 'guardrails enable speed' rather than slowing development?"

A: "Guardrails eliminate all possibility of failure, making development purely predictable" (10 words)
B: "Organizations with strong DORA capabilities see higher productivity gains from AI than organizations without them" (14 words) ✓ CORRECT
C: "Speed comes from guardrails, not from AI; organizations can be fast without AI" (13 words)
D: "Guardrails are purely psychological; developers feel safer even if mistakes aren't prevented" (12 words)

Spread: 10-14 = 4 words ✗ Actually FAIL (exceeds ±3)
```

**Revised Version (PASS):**
```
A: "Guardrails prevent all failures; development becomes completely predictable and safe" (10 words)
B: "Strong DORA capabilities enable 28% higher productivity gains from AI adoption" (11 words) ✓ CORRECT
C: "Speed comes from guardrails alone, not from AI tools or frameworks" (11 words)
D: "Guardrails are just psychological comfort without real safety or actual benefits" (11 words)

Spread: 10-11 = 1 word ✓ PASS
```

---

## Checklist Before Handoff

- [ ] Validated word count for all 50 questions (not spot-check, ALL)
- [ ] Flagged all questions with spread > 3 words
- [ ] Rewritten all failing questions to achieve parity
- [ ] Verified correct answer distribution (no length bias)
- [ ] Created distribution analysis report
- [ ] Tested: Can someone select longest option for 50 questions and achieve 25%+ (random) accuracy? If yes, still FAIL
- [ ] Final validation: Longest option correct in 20-30% of questions (distributed, not biased)
- [ ] Documented validation in quiz file header or separate validation report
- [ ] Submitted to human review with validation report attached

---

## Why This Matters

From educational measurement research (Haladyna et al., 2002):

- **Unequal option lengths:** 60-70% of test-takers can achieve passing scores by selecting longest/shortest without reading
- **Equal length options:** Guessing accuracy drops to 25% (random chance for 4-option MC)
- **Result:** Length parity increases test validity by 45-50%

Your quiz should measure understanding of Chapter 2 concepts, not reading strategy. Strict option length validation ensures that happens.
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 323kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall