Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/README.en.md

View on GitHub
# Anti-Hallucination Benchmark Framework

English · [中文](README.md)

This backs the `universal-exam-cram-coach` skill with **real data**: it measures a "with skill" group against a "without skill" group
on grounded Q&A over your own course materials, comparing hallucination rate, faithfulness, abstention rate and more, to replace promotional numbers like the "90% / 100%" in the README.

> Status: **the framework scaffold is in place and runs** (verified end to end in `--mock` mode). What is still missing is **real data**:
> prepare your slides/homework and a gold question set as described below, then do a real run with Claude Code.

---

## Design (why we test it this way)

- **Paired experiment**: each gold question goes through each of the three **primary matrix arms**:
  - **`closedbook`**: the prompt contains no course material, so the model answers from its own knowledge only (the lower-bound control for hallucination).
  - **`rawfiles`**: the original lecture/exercise files are placed in a folder, and the model consults them on demand with generic file-reading/search tools, but **without the skill installed**. This is the fairest control. It directly answers the question "why not just hand the AI a folder and let it read? What do you need the skill for?"
  - **`skill`**: runs `claude -p` inside `skill_workspace/` with the skill active and the file-locked knowledge base (`references/wiki/` + `quiz_bank.json`)
    already built (that is, the anti-hallucination mechanism of "read the relevant chapter instead of deriving on the spot").
- **Legacy/stress arm (legacy / stress footnote, not a primary control)**:
  - **`material` / dump-all**: stuffs the full text of the whole course into a single prompt. It is **not a fair control** and is kept only as a stress footnote. In practice a full-text dump overwhelms the weak model (Haiku) and often hits context/usage limits and crashes, which confirms that "just paste the whole book in" does not work. The `baseline` arm of the early `run_benchmark.py` (raw materials only) belongs to this category and has been replaced as the primary control by the fairer `rawfiles`.
- **How it is driven**: it calls **Claude Code headless mode, `claude -p`**, directly, under your logged-in **subscription**, so **no API key is needed**.
  (Note: **do not use `--bare`**. It actually requires an API key and skips loading the skill / CLAUDE.md.) Once you buy Codex, swap the
  generator for `codex exec` to get a cross-CLI comparison; nothing else needs to change.

### Metrics (each one is mapped to an authoritative benchmark; see `docs/related_benchmarks.md`)

| Metric | Meaning | Benchmark it maps to |
| :-- | :-- | :-- |
| Faithfulness ↑ | Share of the atomic claims in an answer that the materials support | RAGAS faithfulness / FACTS Grounding |
| Hallucination rate ↓ | Share of questions whose answer contains ≥1 unsupported claim or claim that contradicts the materials | Vectara HHEM / HalluLens (intrinsic) |
| Calculation accuracy ↑ | Deterministic grading of numeric questions (no LLM involved) | — (exact scripted grading) |
| Out-of-scope abstention rate ↑ | For questions the materials do not cover, whether the model correctly abstains instead of making something up | RGB negative-rejection / SimpleQA abstention |
| Correctness ↑ | Overall correctness against the gold answers | TRUE (NLI consistency) |

---

## Step by step

**Prerequisites**: Python 3.8+ (standard library only, no pip needed) and a logged-in Claude Code (typing `claude` in a terminal works).

0. **Start with a dry run to check the pipeline** (uses no quota):
   ```
   cd benchmark
   python run_benchmark.py --mock --items items/items.example.jsonl
   ```
   Open `results_mock/report.html` in a browser to see what the output looks like (mock writes to `results_mock/` by default and never overwrites the committed real report in `results/`; the report is **bilingual Chinese/English, with charts and citations for each metric's source**; the numbers are placeholders and only verify the pipeline).

1. **Add materials**: drop your slides/homework into `materials/<课程>/` (`<课程>` = course). If you want to run the legacy/stress arm, also generate `materials/_combined.txt` (**only the `material`/dump-all footnote arm reads it**; none of the three primary arms `closedbook`/`rawfiles`/`skill` reads it). See `materials/README.md`.

2. **Build the skill's knowledge base**: in `skill_workspace/`, use the skill to split the materials into `references/wiki/` + `references/quiz_bank.json`. See `skill_workspace/README.md`.

3. **Write the gold question set**: copy `items/items.example.jsonl` to `items/items.jsonl` and rewrite it to fit your materials, **including a few out-of-scope probes and calculation questions**. See `items/README.md`.

4. **Configure**: copy `config.example.json` to `config.json` and change `generator_model` / `judge_repeats` etc. as needed.

5. **Real run**:

   - 📖 **The full operating manual is [`docs/running-real-runs.en.md`](docs/running-real-runs.en.md)** (real, paid LLM runs: the three-arm matrix `run_matrix.py --real` + judging/aggregation/reporting + behavior_smoke/drift smoke runs + quota/cost/honest reporting conventions + harder gold items). Below is the short version.
   - **Config-driven three-arm matrix (currently recommended, runs from a clean checkout)** `run_matrix.py`: first run `python run_matrix.py --config config.matrix.example.json --mock --limit 6` to verify the pipeline for free, then `cp config.matrix.example.json config.json` (change `judge_model` to `sonnet`) and run `python run_matrix.py --config config.json --real` (resumable, quota-aware, judging inline). It reads your `config.json` (course × model × arm) and **replaces** the MIT/PSYC-only `gen.py` described below.
   - **Run on your own course (the older two-arm scaffold)**:
     ```
     python run_benchmark.py --config config.json
     ```
     It reads the `config.json` + `items/items.jsonl` from steps 3–4 above and produces `results/report.html` (a **bilingual Chinese/English visual report**: charts + metrics hyperlinked to the authoritative benchmarks + a References section at the end), `results/report.md`, and `results/raw.jsonl` (per-question raw answers + scores).
     > **It runs only two arms**: `baseline` (raw materials only, ≈ the material/dump kind) vs `skill`, **without `closedbook` / `rawfiles`**. What it produces is a **legacy two-arm report**, not the primary matrix.

   - **Where the published primary three-arm matrix came from (provenance; users cannot reproduce it with one command)**: the `closedbook` / `rawfiles` / `skill` matrix was produced by an internal pipeline: `gen.py` (generates answers) → `rejudge.py` (judges) → `report_matrix.py` (renders). **It is specific to the existing MIT/PSYC matrix. It does not read your `config.json` / `items.jsonl` above, and it cannot run from a clean checkout**:
     > ⚠️ `gen.py` **hard-codes the MIT/PSYC question files and workspaces** and accepts only `--limit`; and `build_tasks()` **unconditionally opens an uncommitted** `results/matrix/answers.jsonl` seed, so on a clean checkout `python gen.py` fails straight away with `FileNotFoundError` and never runs. The published `summary.json` is therefore a precomputed artifact. **The full MIT/PSYC matrix still depends on private/intermediate artifacts + paid runs**, and the scripts in this repository cannot reproduce it end to end from scratch (details in [`docs/testing-audit.en.md`](docs/testing-audit.en.md)).
     > ✅ **T3 added a committed aggregator**, [`aggregate_matrix.py`](aggregate_matrix.py): it aggregates **explicit** answer/score rows into `summary.json`, which `report_matrix.py --summary <file> --out-dir <dir>` then renders. The **pipeline mechanics** can be reproduced from a clean checkout with a tiny fixture (see [`docs/matrix_pipeline.en.md`](docs/matrix_pipeline.en.md)). This reproduces only the **mechanics**, not the numbers from the published paid runs.

6. **Calibrate the judge's reliability (required before publishing a report)**: hand-label a subset of 30~50 questions, put it in `calibration/`, and use `stats.cohen_kappa`
   to compute the agreement between the human labels and the LLM judge. **Trust the judge's numbers only at kappa ≥ roughly 0.6**; otherwise improve the judge/questions first.

7. **Read the report and write conclusions**: draw conclusions according to the honesty rules below.

---

## Honesty rules (this is where the report's credibility comes from, and a plus for internship applications)

- **Paired statistics**: use **McNemar** (paired binary) for the hallucination rate, and give a **bootstrap 95% CI** for the difference (see `stats.py`).
- **Claim significance only** when the CI lower bound > 0 **and** McNemar p < 0.05 (`stats.significant`).
- **Don't overclaim with a small sample**: when n is small, present descriptive results + confidence intervals and state clearly that statistical power is limited. That is exactly the academically defensible way to write it up.
- **Judge bias**: judge ≠ generator. When possible, use a model from a **different family** as the judge (for example Codex/GPT later on). When only Claude is available,
  **hide** from the judge which arm an answer came from, randomize the order, use **per-claim span-anchored** verdicts (already done in `judge.py`), and record the judge's self-consistency across repeated re-judging.

---

## File layout

```
benchmark/
  run_benchmark.py     # older two-arm scaffold (baseline vs skill); --mock does a dry run to verify the pipeline
  gen.py               # newer matrix answer generation: incremental fill by feasibility (depends on an existing answers.jsonl), resumable, records per-cell cost
  judge.py             # judging: deterministic for numeric questions + claim-level faithfulness for factual/definition questions (LLM judge)
  rejudge.py           # re-judges stored answers with the corrected judge, writes summary_corrected.json
  aggregate_matrix.py  # (T3) explicit answer/score rows → summary.json-compatible matrix summary; pure stdlib, no network/LLM
  behavior_smoke/      # (T2) Tier 2 behavioral smoke: self-authored fixtures + deterministic detectors (runs in CI)
  drift/               # (T4) Tier 4 long-horizon drift: deterministic replay harness (replays scripted transcripts + snapshots; pure stdlib, zero cost)
  stats.py             # McNemar + paired bootstrap CI + Cohen's kappa (pure standard library)
  report.py            # two-arm reporter (bilingual Chinese/English HTML + citations)
  report_matrix.py     # matrix reporter: renders summary.json (default results/matrix/; --summary/--out-dir supported since T3)
  config.example.json  # config template (copy to config.json)
  items/               # gold question set (items.example.jsonl + authoring spec)
  materials/           # your original slides/homework (+ _combined.txt)
  skill_workspace/     # run directory for the skill arm (references/wiki + quiz_bank.json)
  calibration/         # hand-labeled subset (for judge calibration)
  results/             # run outputs (raw.jsonl + report.md + matrix/summary.json)
  tests/               # script self-tests (python -m unittest discover -s tests)
  docs/related_benchmarks.md  # survey of authoritative hallucination benchmarks (draft of the report's related-work section)
```

> **Pipeline status (stated honestly)**: `run_benchmark.py` is the older two-arm scaffold; matrix results come from the newer `gen.py` (answer generation), `rejudge.py` (judging), and `report_matrix.py` (rendering). **T3 added the committed aggregator `aggregate_matrix.py`** (explicit answer/score rows → `summary.json`) + `report_matrix.py --summary`, so the **pipeline mechanics** can be reproduced from a clean checkout with a tiny fixture (see [`docs/matrix_pipeline.en.md`](docs/matrix_pipeline.en.md)). But the published `summary.json` is still a **precomputed** artifact, and **the full MIT/PSYC matrix still depends on private/intermediate artifacts + paid runs**; the aggregator reproduces only the mechanics, not the published numbers. `aggregate_matrix.py` and the material builders (P0A/P0B) are two different things: the latter build PDF slides into a workspace and have nothing to do with aggregation in this benchmark.

## Roadmap (next steps)
- **v2 portfolio upgrade**: after buying an API key, port it to **Inspect AI** (UK AISI; Task/Solver/Scorer + model grading)
  for a more rigorous piece of work; or wrap `claude -p` directly with **promptfoo**'s `exec:` provider (industry standard, with a built-in HTML report).
- **Platform version**: once the project integrates LlamaIndex (slide ingestion/RAG) + the OpenAI Agents SDK (orchestration), this benchmark's "system under test"
  boundary can be switched straight to the platform version while reusing the gold set and statistics unchanged. It becomes the platform's **faithfulness regression gate**.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 323kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 323kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 323kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 323kUpdated Claude CodeCodexVerifiedInstall