Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/testing-walkthrough.en.md

View on GitHub
# Testing Walkthrough: What This Anti-Hallucination Benchmark Measures and How

English · [中文](测试流程详解.md)

> Written for first-time readers: a plain, step-by-step explanation of everything under `benchmark/`. After reading it you should be able to answer three questions:
> **What are we trying to prove? What data do we prove it with? How are the numbers computed, and why can they be trusted?**
>
> Companion files: the overview is in [`../README.en.md`](../README.en.md), metric sources are in [`related_benchmarks.en.md`](related_benchmarks.en.md), and data sources are in [`materials_sources.en.md`](materials_sources.en.md).

---

## 0. The one-sentence overview

Using **data from real public courses**, we want to test one conclusion:

> Give an AI this exam-prep skill (course materials are first built into a "knowledge base / wiki", and at answer time the AI **retrieves on demand** instead of making things up from memory),
> and on "questions grounded in your own course materials", **does it really fabricate less and more honestly admit "the materials don't cover this"?**

Anyone can write promotional numbers like the "90% / 100%" you see in a README. The point of this benchmark is to **replace promotion with evidence, using reproducible experiments and statistics.**

---

## 1. Start with intuition: three "exam formats"

The core of the whole test is to have **the same set of questions** answered under three different "exam formats", then compare which one answers better and which one fabricates less. These three formats are the **three primary matrix arms**:

| Arm | Code name | Plain analogy | What actually happens |
| :-- | :-- | :-- | :-- |
| **Closed book** | `closedbook` | No book; rely only on what you remember | The prompt contains **no course material at all**; the model answers from its own knowledge only |
| **Raw files + generic agent** | `rawfiles` | A pile of handouts dumped on the desk; flip through them yourself | The original lecture and homework files go into a folder, and the model uses generic file-reading/search tools to **look things up as needed, without this skill installed** |
| **skill (lazy retrieval)** | `skill` | Open book, plus a TA who knows how to use the table of contents | The course materials are first built into a `references/wiki/` knowledge base, and the model **retrieves the relevant chapters as needed** before answering |

`rawfiles` is **the fairest control**. It directly answers the question "why not just hand the AI a folder and let it read? What is the skill for?" The gap between `skill` and `rawfiles` is the skill's real added value.

> **Legacy / stress arm (a stress footnote, not a primary control): `material` / dump-all.**
> Early on there was also a "give it all the material" control arm (`material`, which **stuffs the full text of the entire course into the prompt in one go**). It is **not a fair control** and is now kept only as a **stress footnote**:
> **In practice, dumping the full text of all 20 lectures on a weak model (Haiku) overwhelmed it and made its answers worse**, and the oversized prompts often hit context or usage limits and **produced no answer at all**. That confirms "paste in the whole book" is not workable; it is not an option worth listing side by side. The early `baseline` in `run_benchmark.py` (raw material only) belongs to the same category and has been replaced by the fairer `rawfiles`.

---

## 2. Key materials: the "gold set" and "out-of-scope probes"

To score anything, you first need **reference answers**. For each course we hand-built a **gold set** of questions; each question is one line of JSON:

```json
{"id": "psyc_01",
 "question": "According to Lecture 2, which Nobel Prize-winning biologist described ...?",
 "gold_answer": "Francis Crick",
 "supporting_span": "the Nobel Prize winning biologist, Francis Crick, described as \"The Astonishing Hypothesis.\"",
 "source_file": "materials/psych_yale_psyc110/lecture02.md",
 "answer_type": "factual",
 "answerable": true}
```

Field by field:

- **`question`**: the question text (English questions match the original English lectures, which avoids translation loss).
- **`gold_answer`**: the reference answer.
- **`supporting_span`**: **the original passage in the lecture that this question's answer comes from**. Grading uses only this passage as its basis; this is what keeps the grading from being arbitrary.
- **`answer_type`**: `factual` / `definition` / `numeric` (calculation questions, which are graded deterministically).
- **`answerable`**: **whether the question can be answered**.

### Out-of-scope probes: `answerable: false`

This is the design that **best shows the anti-hallucination goal** of the whole test. We deliberately mix in a batch of questions **whose answers are simply not in the materials**, for example:

```json
{"id": "psyc_p01",
 "question": "How many students were enrolled in PSYC 110 when it was recorded?",
 "gold_answer": "", "answerable": false}
```

For a question like this, **the only correct behavior is to say honestly "not covered in the materials"**.
- If the model honestly abstains → it earns credit (it did not make anything up).
- If the model invents a number anyway → **that is a hallucination**, and it loses points.

> For the 6.006 course: 65 questions = **55 answerable questions + 10 out-of-scope probes**.

### Two item sets: with answers (local) vs. questions only (blind test / publishable)

- `items_*_full.jsonl`: the complete gold set **with reference answers**. It **does not go into the public repository** (it may quote copyrighted lectures verbatim).
- `items_*_full_q.jsonl`: the version with **questions only and no answers**, used when feeding the models under test. This guarantees that the model "really answers" instead of "copying the answer", and this file can be published.

---

## 3. The five metrics: what each one measures (plain version)

| Metric | Direction | In plain words | Strict definition |
| :-- | :-- | :-- | :-- |
| **Correctness** | ↑ higher is better | How many questions it got right | Overall proportion correct against the gold set |
| **Faithfulness** | ↑ | How much of what it says has a source | After the answer is split into atomic claims, the proportion **supported by `supporting_span`** |
| **Hallucination rate** | ↓ lower is better | Share of questions where it "just made something up" | Share of questions containing ≥1 **unsupported or material-contradicting** claim |
| **Out-of-scope abstention rate (OOS)** | ↑ | Among questions it cannot answer, the share where it **honestly says it doesn't know** | Proportion of correct abstentions on the out-of-scope probes |
| **Multi-round convergence** | ↑ | Does a more complete knowledge base lead to better answers? | How the skill arm's correctness changes as the wiki gains chapters |

Every metric is **aligned with an established industry benchmark** (FACTS Grounding, Vectara HHEM, RAGAS, RGB, and others; see [`related_benchmarks.en.md`](related_benchmarks.en.md)), so the numbers are not definitions we made up ourselves.

---

## 4. Grading (the judge): how a number is extracted from an answer

All grading logic lives in [`../judge.py`](../judge.py), and it handles three cases:

### (a) Out-of-scope probe → did it abstain?
It checks whether the answer contains an **abstention marker** (`ABSTAIN_MARKERS`) such as 「材料中未涵盖 / not covered / 无法确定」 ("not covered in the materials" / "not covered" / "cannot determine").
Abstaining = correct, zero hallucination; answering anyway = one hallucination. **No LLM is involved at any point; the grading is deterministic.**

### (b) Calculation question (`numeric`) → deterministic numeric comparison
It extracts **the last number** in the answer and compares it with `gold_answer` within a tolerance (`check_numeric`).
**No LLM here either**: the true value is exact, so there is no reason to introduce judge noise.

### (c) Factual/definition question → claim-by-claim LLM judge
This is the only place an AI judge is used, and it has several layers of anti-bias design:
1. **Split into atomic claims**: the judge splits the answer into individual "minimal factual statements" and decides, one by one, **whether each can be inferred from `supporting_span`** (1 = yes / 0 = no or contradicted). `faithfulness = supported claims / total claims`.
2. **Use only the evidence, not its own knowledge**: the prompt explicitly requires the judge to decide **based only on the given span**, which rules out "the judge also remembers the answer, so it goes easy".
3. **Blind judging**: the judge **does not know which arm an answer came from** (closed book? skill?), which avoids bias.
4. **Repeated judging + majority vote**: each question is judged `judge_repeats` times and the majority wins; the judge's **self-consistency** `judge_self_consistency` is also recorded (whether repeated judgments agree, which shows how stable the judge is).
5. **Judge ≠ generator**: where possible, a model from a **different family** is used as the judge. This round of 6.006 used **Sonnet** as the judge (the models under test were Opus/Sonnet/Haiku, so this point is not yet perfect; see the honest limits in Section 9).

---

## 5. Statistics (stats): why it isn't just "take an average and call it done"

The easiest mistake to make with a small sample is "seeing 72% > 70% and declaring that the skill wins". [`../stats.py`](../stats.py) uses three tools to block that kind of overclaiming:

- **McNemar test** `mcnemar()`: because the same question goes through every arm, the data are **paired**, and only the questions where "the two arms answered differently" carry information. It tests whether "the skill hallucinates less than the control" is statistically real rather than luck.
- **Paired bootstrap confidence interval** `paired_bootstrap_ci()`: gives a 95% interval for the "difference", telling you how wide the uncertainty around the improvement is.
- **Cohen's kappa** `cohen_kappa()`: measures agreement between "human grading vs. the AI judge", and is **a gate that must be passed before the judge's numbers can be trusted**.

**The hard rule for significance** (`significant()`): only when **the lower bound of the confidence interval > 0** and **McNemar p < 0.05** do we dare to say "significant". Otherwise we honestly write the result up as "descriptive + interval + limited sample", **which is exactly the academically defensible way to write it**.

---

## 6. The complete end-to-end flow (7 steps)

This is the full flow for actually "running the test once":

```
1. Prepare materials ──→ 2. Real ingest builds the wiki ──→ 3. Write gold set (incl. OOS probes) ──→ 4. 3 arms × each model, blind
                                                                                                                   │
        7. Final report ←── 6. Multi-round feedback loop ←── 5. Grading (deterministic numeric + blind judge + human kappa)
```

1. **Prepare the complete course materials**: put all the lectures of one public course into `materials/<course>/` (copyrighted content, **not in the public repository**).
2. **Run a real ingest to build the knowledge base**: the skill's `scripts/ingest.py` splits the lectures into `skill_workspace/.../references/wiki/ch01..N.md`; this is what the skill arm actually retrieves from.
3. **Write the gold set**: hand-write 40–55 answerable questions (covering every lecture) + 10 out-of-scope probes, store them as `items/items_*_full.jsonl`, and generate the questions-only `_q.jsonl`.
4. **Three-arm blind test**: every question is answered once under each of the three arms by each model (Opus/Sonnet/Haiku). The runs are driven by **Claude Code headless mode `claude -p`**, using your logged-in **subscription** identity, so **no API key is needed** (the three-arm × multi-model matrix is generated cell by cell by [`gen.py`](../gen.py) and can resume after an interruption; [`run_benchmark.py`](../run_benchmark.py) is the early two-arm scaffold).
5. **Grading**: score each answer with the three grading paths from Section 4.
6. **Multi-round feedback loop**: fill in the skill's knowledge base step by step, from "incomplete" to complete (wiki 7 chapters → 14 chapters → 20 chapters), and see whether correctness climbs along with it. This demonstrates the causal link "the more complete the knowledge base, the more accurate the answers".
7. **Final report**: all numbers are compiled into a bilingual Chinese/English HTML report + comparison charts.

> **Want to see what the pipeline looks like before spending any quota?** Use mock (placeholder answers and fake numbers, purely to verify the flow):
> ```
> cd benchmark
> python run_benchmark.py --mock --items items/items.example.jsonl
> ```
> Then open `results_mock/report.html` in a browser (mock writes to `results_mock/` by default and never overwrites the committed real report `results/report.html`). For a **real run**, drop `--mock` and set up `config.json`.

---

## 7. Generation, grading, reporting: what produces data and what only renders

**Generation paths (what actually runs the models and produces answers):**
- [`run_benchmark.py`](../run_benchmark.py): the older **two-arm scaffold** (baseline vs skill); `--mock` does a dry run to verify the pipeline.
- [`gen.py`](../gen.py): the newer **matrix answer generation** path. It generates answers per `course × model × arm`, **can resume after an interruption, and records the cost of each cell** (it writes `results/matrix/gen_answers.jsonl`). Note that it is **an incremental gap-filler ordered by feasibility** (rawfiles for both courses + PSYC closedbook/skill + a rerun of only the algo material cells that errored + PSYC material) and **depends on an existing `answers.jsonl`**. It does not rebuild the whole matrix from scratch, so the published matrix cannot be reproduced from scratch using only the scripts in this repository.
- [`rejudge.py`](../rejudge.py): uses the corrected `judge.py` to **re-grade stored answers** and writes `results/matrix/summary_corrected.json`.

**Reporters (render only, compute nothing):**
- [`report.py`](../report.py): the early **two-arm** (baseline vs skill) reporter; produces `results/report.html`.
- [`report_matrix.py`](../report_matrix.py): the matrix reporter; produces `results/matrix/report.html` + a set of standalone comparison-chart SVGs (one each for correctness, hallucination rate, out-of-scope abstention, and convergence, each in a Chinese and an English version). The primary arm definitions (`closedbook` / `rawfiles` / `skill`) live in its `ARMS` constant.
- [`rounds.py`](../rounds.py): draws the **multi-round convergence line chart** (`results/convergence.html`), which shows the effect of step 6, "fill in the knowledge base round by round".

The "single source of truth" for every number is `results/matrix/summary.json`; the reporters only render it into charts.

> **Honesty note**: `report_matrix.py` only **renders** `summary.json` and computes nothing itself. The committed aggregator [`aggregate_matrix.py`](../aggregate_matrix.py), which builds `summary.json` from **explicit** answer/score rows, has now been added (together with the general three-arm runner [`run_matrix.py`](../run_matrix.py)), and the **pipeline mechanism** can be reproduced from a clean checkout with a tiny fixture (see [`matrix_pipeline.en.md`](matrix_pipeline.en.md)). But the published `summary.json` is still a **precomputed** artifact: the aggregator reproduces the mechanism, not the published numbers from the paid runs.

---

## 8. Real results at a glance (MIT 6.006 Algorithms, n=65, graded by Sonnet)

The data come from `results/matrix/summary.json`. How to read them: **for each model, compare its performance across the arms.** Grading was done by Sonnet, numeric questions were compared exactly by the program, and the grading was calibrated with a human spot check of 16 questions (see Section 9, Cohen's kappa = 0.875).

> **Note (arm definitions)**: the control column in the early at-a-glance tables below is the **legacy/stress arm "Full material" (`material`)**, not a primary control arm. The current **three primary arms are `closedbook` / `rawfiles` / `skill`**; the complete three-arm × three-model matrix, including `rawfiles` (raw files + generic agent), is in [`../README.en.md`](../README.en.md) and [`../results/matrix/report.html`](../results/matrix/report.html). The numbers here are kept as they were for the record and serve only as a historical quick read.

**Correctness (↑)**

| Model | Closed book | Full material¹ | skill |
| :-- | :-- | :-- | :-- |
| Opus | 16% | 88% | **91%** |
| Sonnet | 49% | 93% | **85%** |
| Haiku | 18% | —² | **93%** |

¹ material is scored only on real answers (rate-limit errors excluded): Opus n=50, Sonnet n=27. ² For Haiku, only 2 of the 65 material questions ran successfully; the sample is too small to list.

**Out-of-scope abstention rate (↑, the share of out-of-scope questions where the model honestly says "I don't know")**

| Model | Closed book | Full material | skill |
| :-- | :-- | :-- | :-- |
| Opus | 60% | 100% | **100%** |
| Sonnet | 50% | —² | **100%** |
| Haiku | 60% | —² | **100%** |

**How to read these two tables (this is the conclusion):**
1. **The skill arm has the highest correctness for all three models (85–93%), and every one is above closed book.**
2. **"Full material" is not a good approach**: **when it does run**, its correctness is comparable to skill (Opus 88%, Sonnet 93%), but stuffing an entire course (~100K tokens) into the prompt **frequently exceeds how much the model can process in one go and triggers usage limits**. On Haiku, only **2 of the 65 questions ran successfully** (the rest could not return a valid answer and so were not scored). This engineering infeasibility is exactly the practical value of the skill's **chapter-based on-demand retrieval** (each query is short, stable, and cheap).
3. **Out-of-scope abstention: skill is at 100% on all three models** (zero fabrication on questions the materials do not cover).
4. **Read the hallucination rate honestly**: the skill's claim-level hallucination rate of 20–29% is not the lowest, because strict grading also counts "correct, but adds details the supporting passage does not mention" as a hallucination. That is not the same as fabrication (for true fabrication, look at the out-of-scope probes, where skill = 100% honest).

**Multi-round convergence**: as the skill arm's knowledge base is filled in from 7→14→20 chapters, correctness goes **28% → 62% → 87%**. The more complete the knowledge base, the more accurate the answers; the causal chain is clear.

**Another course: Yale PSYC 110 (humanities, real runs, three models)** closed book vs skill correctness: Opus **60%→98%**, Sonnet **55%→92%**, Haiku **55%→80%**; out-of-scope abstention for skill is **100% on all three models** (closed book 80–90%). The humanities course shows the same large advantage. (PSYC's material arm, at ~240K tokens, exceeds the context window and cannot run, which itself confirms that "dumping the whole course" is not workable.)

---

## 9. Honest limits (what cannot be treated as settled yet)

This is the most valuable section of any experiment. **Known limitations** of the current results:

- **Grading has been independently calibrated by humans twice**: ① a spot check of 16 questions, comparing human and judge results question by question, gave Cohen's kappa = **0.875**; ② another **24 questions** were labeled with B5's general calibration methodology (the same flow as `calibrate_matrix`: **four strata** of answerable judged correct/incorrect + out-of-scope abstained/did not abstain, with the judge's verdict **hidden** from the human, and quota-error answers filtered out first), giving 91.7% agreement and Cohen's kappa = **0.833**. Both rounds show high agreement (≥0.8) and corroborate each other. The human–judge disagreements observed in both rounds were **all cases of the judge being too strict** (a correct answer judged wrong; for example, a spelling variant of Extraversion on the OCEAN question was marked wrong). This is a sampling signal (not proof about every judgment), and it suggests the numbers lean conservative rather than inflated. **Grading still uses only a single model family (Sonnet)**, which is a known limitation.
- **The hallucination-rate metric is strict**: claim-level strict grading penalizes answers that are "correct but thoroughly elaborated" (see above), so the hallucination rate is not the skill's strong suit; its strengths are correctness + out-of-scope abstention.
- **The samples are small**: 6.006 n=65, PSYC n=50, so statistical power is limited. Results are presented as "descriptive + interval", without forcing claims of significance.
- **A very small number of judge_error cases remain after re-grading** (~9/495 LLM judgments still come back empty and are conservatively counted as incorrect).

> These are not "hidden flaws" but **deliberately stated limits**. A report that spells out "under what conditions my number holds" is far more credible than one that just shouts "100%".

---

## 10. Privacy / copyright boundary: what goes into the repository and what does not

This is enforced by [`../.gitignore`](../.gitignore):

- **Never in the public repository** (kept locally): the original copyrighted lecture text (MIT/Yale transcripts), the complete gold sets with answer keys `items_*_full.jsonl`, the raw per-question logs `raw*.jsonl` / `answers` / `scores`, the knowledge bases generated by the skill, and the human calibration labels.
- **May be committed**: the questions-only gold sets `_q.jsonl`, the aggregate summary `summary.json`, the report `report.html` and the comparison-chart SVGs, and all test code.

The principle: **anything that supports reproduction and backs up the claims is public; anything that might infringe copyright or leak answers stays local.**

---

## 11. File map (one line to remember each file by)

```
benchmark/
  run_benchmark.py     # older two-arm scaffold (baseline vs skill); --mock does a dry run to verify the pipeline
  gen.py               # newer matrix answer generation: incremental gap-filling by feasibility (depends on existing answers.jsonl), resumable, records per-cell cost
  judge.py             # grading: OOS checks abstention / numeric is deterministic / factual uses a claim-by-claim blind LLM judge
  rejudge.py           # re-grades stored answers with the corrected judge → summary_corrected.json
  stats.py             # paired statistics: McNemar + bootstrap confidence interval + Cohen's kappa
  report.py            # two-arm reporter (bilingual Chinese/English HTML + citations)
  report_matrix.py     # matrix reporter: only renders the precomputed summary.json (primary arms closedbook/rawfiles/skill)
  rounds.py            # multi-round convergence line chart (effect of filling in the knowledge base round by round)
  items/               # gold sets (_full with answers = local; _q questions only = publishable)
  materials/           # original course materials (copyrighted, all local)
  skill_workspace/     # working directory for the skill arm (references/wiki + quiz_bank)
  calibration/         # human-labeled subset (for judge kappa calibration)
  results/             # run outputs (summary.json + reports + comparison charts)
  tests/               # script self-tests: cd benchmark && python -m unittest discover -s tests
  docs/                # methodology and citations (this file + metric sources + data sources)
```

---

## 12. The one-sentence summary

> **Same questions, three exam formats, blind-judge grading, paired statistics as the safety net, and out-of-scope probes aimed squarely at catching fabrication.**
> The conclusion rests not on adjectives but on **reproducible numbers + honesty about stated limits**. That is how an "anti-hallucination" claim ought to be backed up.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall