Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/README.md

View on GitHub
# 防幻觉 Benchmark 框架

[English](README.en.md) · 中文

给 `universal-exam-cram-coach` 这个 skill 做**真实数据背书**:用实测对比"装了 skill / 没装 skill"
两组在"基于你自己课件的有据问答"上的幻觉率、忠实度、弃答率等,替代 README 里"90% / 100%"那种宣传数字。

> 现状:**框架脚手架已就绪并可跑**(`--mock` 模式端到端验证过)。还差**真实数据**——
> 你需要按下面把课件/作业、金标题集准备好,再用 Claude Code 真跑。

---

## 设计(为什么这么测)

- **配对实验**:同一道金标题,分别过**主对照三臂(primary matrix arms)**——
  - **`closedbook`**:提示里不给任何课程材料,只靠模型自身知识答(防幻觉下限对照)。
  - **`rawfiles`**:把原始讲义/习题文件放进一个文件夹,模型用通用文件读取/检索工具按需查阅,但**不装本技能**。这是最公平的对照——直接回答「丢个文件夹给 AI 自己读不就行了,还要技能干嘛」。
  - **`skill`**:在 `skill_workspace/` 里跑 `claude -p`,skill 已激活、`references/wiki/` + `quiz_bank.json`
    文件锁定知识库已建好(即「读取相关章节而非现场推导」的防幻觉机制)。
- **遗留/压力臂(legacy / stress footnote,非主对照)**:
  - **`material` / dump-all**:把整门课全文一股脑塞进一次提问。它**不是公平对照**,仅保留为压力脚注——实测表明全文 dump 会淹没弱模型(Haiku)、且常触发上下文/用量上限而跑崩,恰好印证「整本贴进去」不可行。早期 `run_benchmark.py` 的 `baseline`(只给原始材料)即属此类,已被更公平的 `rawfiles` 取代为主对照。
- **驱动方式**:直接调 **Claude Code 无头模式 `claude -p`**,用你登录的**订阅**身份,**不需要 API key**。
  (注意:**不要用 `--bare`**——它反而需要 API key,且会跳过 skill / CLAUDE.md 加载。)Codex 买了之后把
  生成器换成 `codex exec` 即可做跨 CLI 对比,其余不用动。

### 测什么指标(每项都对标了权威基准,见 `docs/related_benchmarks.md`)

| 指标 | 含义 | 对标基准 |
| :-- | :-- | :-- |
| 忠实度 faithfulness ↑ | 答案里"能被材料支持的原子论断"占比 | RAGAS faithfulness / FACTS Grounding |
| 幻觉率 hallucination ↓ | 含 ≥1 条无依据/与材料矛盾论断的题占比 | Vectara HHEM / HalluLens(intrinsic) |
| 计算题准确率 ↑ | 数值题确定性判分(不经 LLM) | —(脚本精确判) |
| 越界弃答率 ↑ | 材料没有的题,是否正确地弃答而非编造 | RGB negative-rejection / SimpleQA 弃答 |
| 正确率 correctness ↑ | 对金标的整体正确率 | TRUE(NLI 一致性) |

---

## 一步步怎么做

**前置**:Python 3.8+(标准库即可,无需 pip)、Claude Code 已登录(终端敲 `claude` 能用)。

0. **先空跑验证流程**(不花额度):
   ```
   cd benchmark
   python run_benchmark.py --mock --items items/items.example.jsonl
   ```
   用浏览器打开 `results_mock/report.html` 看看产物长什么样(mock 默认写 `results_mock/`,绝不覆盖已提交的真报告 `results/`;**中英双语、带图表和指标出处引用**;数字是占位的,仅验证管线)。

1. **放材料**:把课件/作业丢进 `materials/<课程>/`;如要跑遗留/压力臂,再生成 `materials/_combined.txt`(**仅 `material`/dump-all 脚注臂读它**;主对照三臂 `closedbook`/`rawfiles`/`skill` 都不读它)。见 `materials/README.md`。

2. **建 skill 知识库**:在 `skill_workspace/` 里用 skill 把材料切成 `references/wiki/` + `references/quiz_bank.json`。见 `skill_workspace/README.md`。

3. **编金标题集**:把 `items/items.example.jsonl` 复制为 `items/items.jsonl`,依据你的材料改写,**包含若干越界探针和计算题**。见 `items/README.md`。

4. **配置**:复制 `config.example.json` 为 `config.json`,按需改 `generator_model` / `judge_repeats` 等。

5. **真跑**:

   - 📖 **完整操作手册见 [`docs/running-real-runs.md`](docs/running-real-runs.md)**(真·付费 LLM 跑:三臂矩阵 `run_matrix.py --real` + 判分/聚合/报告 + behavior_smoke/drift 冒烟 + 配额/成本/诚实口径 + 更难金标)。下面是简版。
   - **配置驱动的三臂矩阵(现推荐,从干净 checkout 可跑)** `run_matrix.py`:先 `python run_matrix.py --config config.matrix.example.json --mock --limit 6` 免费验证管线,再 `cp config.matrix.example.json config.json`(改 `judge_model` 为 `sonnet`)后 `python run_matrix.py --config config.json --real`(resumable、配额感知、判分内联)。它读你的 `config.json`(课程×模型×臂),**取代**下面只跑 MIT/PSYC 的 `gen.py`。
   - **用你自己的课程跑(较早的两臂脚手架)**:
     ```
     python run_benchmark.py --config config.json
     ```
     它读上面第 3–4 步的 `config.json` + `items/items.jsonl`,产出 `results/report.html`(**中英双语可视化报告**:图表 + 指标超链接到权威基准 + 末尾 References)、`results/report.md`、`results/raw.jsonl`(逐题原始答案+评分)。
     > **它只跑两臂**:`baseline`(只给原始材料,≈ material/dump 类)vs `skill`,**不含 `closedbook` / `rawfiles`**——产出的是**遗留两臂报告**,不是主对照矩阵。

   - **已发布的主对照三臂矩阵是怎么来的(provenance,非用户可一键复现)**:`closedbook` / `rawfiles` / `skill` 矩阵由内部管线 `gen.py`(生成答案)→ `rejudge.py`(判分)→ `report_matrix.py`(渲染)产出。**它专用于既有的 MIT/PSYC 矩阵,既不读你上面的 `config.json` / `items.jsonl`,也不能从干净 checkout 跑起来**:
     > ⚠️ `gen.py` **硬编了 MIT/PSYC 的题面文件与工作区**、只接受 `--limit`;且 `build_tasks()` 会**无条件打开一份未提交的** `results/matrix/answers.jsonl` 种子——干净 checkout 上 `python gen.py` 会直接 `FileNotFoundError`、根本跑不起来。所以已发布的 `summary.json` 是预先计算的产物,**完整 MIT/PSYC 矩阵仍依赖私有/中间产物 + 付费运行**,本仓库脚本无法从零端到端复现(详见 [`docs/testing-audit.md`](docs/testing-audit.md))。
     > ✅ **T3 已补上提交版聚合器** [`aggregate_matrix.py`](aggregate_matrix.py):从**显式**的 answer/score 行聚合出 `summary.json`,再 `report_matrix.py --summary <file> --out-dir <dir>` 渲染——**流水线机制**用微型 fixture 即可从干净 checkout 复现(见 [`docs/matrix_pipeline.md`](docs/matrix_pipeline.md))。这只复现**机制**,不复现已发布的付费跑数字。

6. **裁判可信度校准(出报告前必做)**:人工标注 30~50 题子集放进 `calibration/`,用 `stats.cohen_kappa`
   算"人工 vs LLM 裁判"的一致性。**kappa ≥ 0.6 左右才信任裁判的数字**;否则先改进裁判/题目再说。

7. **读报告、写结论**:按下面的诚实规则下结论。

---

## 诚实规则(这就是报告可信度的来源,也是给实习的加分项)

- **配对统计**:幻觉率用 **McNemar**(配对二分),差值给 **bootstrap 95% CI**(见 `stats.py`)。
- **显著性只在**:CI 下界 > 0 **且** McNemar p < 0.05 时才声称"显著"(`stats.significant`)。
- **样本小就别硬说**:n 小时按描述性 + 置信区间呈现,并明确说明统计功效有限——这恰恰是学术上站得住的写法。
- **裁判偏置**:裁判 ≠ 生成器。条件允许就用**不同家族**模型当裁判(如以后用 Codex/GPT);只有 Claude 时,
  对裁判**隐藏**答案来自哪一臂、随机化顺序、用**逐条 span 锚定**的判定(已在 `judge.py` 里这么做),并记录裁判多次复评的自一致性。

---

## 文件结构

```
benchmark/
  run_benchmark.py     # 较早的两臂脚手架(baseline vs skill);--mock 可空跑验证管线
  gen.py               # 较新的矩阵答案生成:按可行性增量补齐(依赖既有 answers.jsonl)、可断点续跑、记录每格成本
  judge.py             # 判分:数值题确定性 + 事实/定义题 claim 级忠实度(LLM 裁判)
  rejudge.py           # 用修正后的 judge 重判已存答案,写 summary_corrected.json
  aggregate_matrix.py  # (T3) 显式 answer/score 行 → summary.json 兼容矩阵 summary;纯标准库、无网络/LLM
  behavior_smoke/      # (T2) Tier 2 行为冒烟:自撰 fixture + 确定性探测器(进 CI)
  drift/               # (T4) Tier 4 长程漂移:确定性 replay harness(回放脚本化 transcript + 快照;纯 stdlib、零成本)
  stats.py             # McNemar + 配对 bootstrap CI + Cohen's kappa(纯标准库)
  report.py            # 两臂报告器(中英双语 HTML + 引用)
  report_matrix.py     # 矩阵报告器:渲染 summary.json(默认 results/matrix/;T3 起支持 --summary/--out-dir)
  config.example.json  # 配置模板(复制为 config.json)
  items/               # 金标题集(items.example.jsonl + 编写规范)
  materials/           # 你的原始课件/作业(+ _combined.txt)
  skill_workspace/     # skill 臂的运行目录(references/wiki + quiz_bank.json)
  calibration/         # 人工标注子集(裁判校准用)
  results/             # 运行产物(raw.jsonl + report.md + matrix/summary.json)
  tests/               # 脚本自测(python -m unittest discover -s tests)
  docs/related_benchmarks.md  # 权威幻觉基准综述(报告的 related work 草稿)
```

> **管线现状(诚实标注)**:`run_benchmark.py` 是较早的两臂脚手架;矩阵结果由较新的 `gen.py` 生成答案、`rejudge.py` 判分、`report_matrix.py` 渲染。**T3 补上了提交版聚合器 `aggregate_matrix.py`**(显式 answer/score 行 → `summary.json`)+ `report_matrix.py --summary`,**流水线机制**用微型 fixture 即可从干净 checkout 复现(见 [`docs/matrix_pipeline.md`](docs/matrix_pipeline.md))。但已发布的 `summary.json` 仍是**预先计算**的产物,**完整 MIT/PSYC 矩阵仍依赖私有/中间产物 + 付费运行**;聚合器只复现机制、不复现已发布数字。`aggregate_matrix.py`/材料构建器(P0A/P0B)是两回事——后者把 PDF 课件建成工作区,与本基准聚合无关。

## 路线(后续)
- **v2 portfolio 升级**:买了 API key 后可移植到 **Inspect AI**(UK AISI,Task/Solver/Scorer + 模型判分),
  作品更"硬";或用 **promptfoo** 的 `exec:` provider 直接包 `claude -p`(业界标准,自带 HTML 报告)。
- **平台化**:当项目接入 LlamaIndex(课件入库/RAG)+ OpenAI Agents SDK(编排)后,本 benchmark 的"被测系统"
  边界可直接换成平台版,金标集和统计完全复用——它就是平台的**忠实度回归门禁**。
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall