ZeKaiNie/universal-examprep-skill / full
universal-exam-cram-coach-full
Verified帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。
benchmark/REPORT.en.md
View on GitHub# A hallucination benchmark: the skill takes 11% to ~99%
[中文](REPORT.md) · English
*July 2026 · ~6 min*
We measure one specific thing: give a model the same question and **only change whether it gets the course materials** — how do correctness and fabrication move? Three conditions — closed-book (nothing), raw files (drop the lecture files in a folder, generic agent retrieves), and with this skill (lectures sliced into a chaptered knowledge base, pulled on demand). All judging is by Sonnet: factual answers first try a verbatim gold match, falling to per-claim checking only if that misses.
The skill's value is **grounding** — connecting what's in your materials but not in the model's head, accurately, and never fabricated. Two real measurements make the case.
---
## In your materials, not in the model: from 11% to ~99%
Earlier versions mined our own questions from the lectures, fairly criticized as not rigorous, so this version targets details **only someone who watched that lecture would know**: from the 20 Yale PSYC 110 lecture transcripts we mined 54 questions — the professor's own examples, personal asides, obscure studies he named, exact numbers (his childhood phone number, ~30 billion cerebellar neurons, a study's 92%), each answer **anchored verbatim to the transcript**. World knowledge can't answer these.
<div align="center"><img src="docs/img/hard_psyc_correct_en.svg" width="620" alt="materials-specific: closed-book vs arms" /></div>
*54 materials-specific questions, judge Sonnet. All three models collapse to **11%–13%** closed-book; hand the materials back (raw files or skill) and correctness returns to **98%–100%** — every model reaches **100% with the skill**.*
Same model, same question — **just changing whether it gets the materials moves correctness from 11% to ~99%**. That's the point of the skill: when the answer is in your materials but not the model's head, it connects the materials **accurately**. A STEM course (6.006 Algorithms, 65 questions) shows the same pattern:
<div align="center"><img src="docs/img/hard_algo_correct_en.svg" width="620" alt="6.006 materials-specific: closed-book vs arms" /></div>
*6.006, 65 questions. Closed-book 31%–58% → with the skill 91% (all three models). One humanities-recall course, one algorithm-reasoning course, one curve.*
The skill matches a "raw files agent" on accuracy but costs less — it pulls only the compressed relevant chapters, while raw files re-scans the whole pile each question ($0.10 vs $0.117 per question on PSYC, $0.063 vs $0.066 on 6.006) — and helps weaker models most.
---
## Not in the materials at all: the skill says "not covered" 100% of the time
Every set seeds **out-of-scope probes** — questions the materials genuinely don't answer (e.g. "how many students enrolled in this course"). The right behavior is to abstain; fabricating is a hallucination.
<div align="center">
<img src="docs/img/oos_psyc_abst_en.svg" width="440" alt="PSYC out-of-scope: honest abstention" />
<img src="docs/img/oos_algo_abst_en.svg" width="440" alt="6.006 out-of-scope: honest abstention" />
</div>
*Honest-abstention rate on out-of-scope probes, higher is better (left PSYC / right 6.006). With the skill (and raw files), **100% across both courses and all three models**; closed-book only 60%–90% (it fabricates a plausible answer).*
This is the skill's most consistent result and the clearest anti-hallucination measure: **faced with a question the materials don't answer, it says "not in the materials" 100% of the time — never fabricates**; closed-book models fabricate a plausible answer 10%–40% of the time.
---
## Correctness rises with knowledge-base coverage
The skill's correctness is driven by coverage. Growing the 6.006 knowledge base from 7 chapters to 14 to 20 raises correctness from 28% to 62% to 87% — monotonic.
<div align="center"><img src="results/matrix/chart_convergence_en.svg" width="560" alt="coverage vs correctness" /></div>
*Knowledge base 7 → 14 → 20 chapters, correctness 28% → 62% → 87%. Coverage is the driver.*
---
## How we judged (and a bug we fixed)
Judging is by Sonnet: numeric questions by exact programmatic comparison; factual questions first by `contains_gold` verbatim match, falling to per-claim checking only if that misses.
Doing this run we found the judge was **too strict** in two ways. **First**, a correct answer that **wrapped the gold in markdown** (`**"a donkey and a big bag of peanuts"**`) or **added correct context** was missed by the verbatim match and then marked wrong as "a claim beyond the evidence." Fix: strip markdown/quotes before matching, and tell the judge explicitly that "extra correct detail doesn't change correctness."
**Second**, on 14 items the judge call itself **crashed and returned no verdict** — which the aggregator counts as wrong by default, penalizing whichever arm they happened to land in. We re-adjudicated all 14 with **three independent judges each, majority vote** (containment / abstention-skeptic / domain-fact lenses; every verdict was unanimous). Eight were genuinely correct answers the crash had hidden — e.g. an answer quoting *"they built my minivan"* against the gold *"they built his minivan,"* or one quoting the LIS subproblem verbatim; the other six were genuine abstentions ("not sure") or wrong commitments ("12" against the gold "ten"), which stay not-correct. After both fixes, **all three models reach 100% with the skill on PSYC**.
Both corrections point the same way as human calibration: in both human spot-checks (16-item kappa = 0.875, 24-item stratified blind kappa = 0.833) every human–judge disagreement was the judge being too strict, so the numbers lean conservative, not inflated.
---
## Why these three arms
- **Closed-book** — no materials, exposes the model's prior-knowledge floor.
- **Raw files + generic agent** — drop the raw lectures in a folder, the model retrieves, **without this skill**. The fairest control — "why not just drop a folder at the AI?"
- **With the skill** — lectures organized into a chaptered knowledge base, chapters pulled lazily.
Raw files is already strong, so the skill wins on "same accuracy for less cost + more help to weaker models + never fabricates," not by blowing it away — reported as-is.
---
## Limitations
- **The skill measures grounding** (content specific to your materials), not the generic knowledge a model already has cold — on that kind of question the model is fine closed-book, and the skill isn't where its value lies.
- **Small sample**: 54 materials-specific PSYC questions, 65 for 6.006; 10 out-of-scope probes per course. Trend evidence, not large-scale statistics.
- **Materials-specific gold held locally** (verbatim transcript quotes, to avoid copyright/leakage); reproducible by holders via `run_matrix.py --real`.
- **The Sonnet judge shares a family with the graded models** — both human kappa passes were high, but a known limitation.
Full commands and reproduction in [`docs/running-real-runs.md`](docs/running-real-runs.en.md); metrics are benchmarked against FACTS Grounding, Vectara HHEM, RAGAS, RGB (see [`docs/`](docs/)).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |