Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/judge-calibration.md

View on GitHub
# 判分校准(B5 · Tier-5)

[English](judge-calibration.en.md) · 中文

裁判(LLM judge)的数字能不能信,要**校准**才知道。本层给出:① 更稳的数值判分;② 通用的
「人工 vs 裁判」Cohen's kappa 校准工具(读 B4 `run_matrix` 输出,任意课程可用);③ 跨家族裁判提醒;
④ near-miss 越界探针的出题建议。**诚实前提**:kappa < ~0.6 时别信任裁判数字,先改裁判/题目。

## 1. 数值判分加固(`judge.check_numeric` / `_extract_final_number`)

旧实现用 `-?\d+(?:\.\d+)?` 抓答案里最后一个数字,会把常见写法判错:

| 答案写法 | 旧结果 | 现结果 |
| :-- | :-- | :-- |
| `1,000,000`(千分位逗号) | 抓成 `000` = 0 → 判错 | `1000000` ✓ |
| `1e6` / `1.5e-3`(科学计数) | 抓成 `6` / `3` → 判错 | `1000000` / `0.0015` ✓ |
| `10^6` / `2^2.5`(乘方,指数可小数) | 抓成 `6` → 判错 | 先算成 `1000000` / `5.657` ✓ |
| `50%` / `$8 KB`(带符号/单位) | 视情况 | 取数值 `50` / `8` ✓ |
| `1.234,56`(欧式)/ `3,14`(乱逗号) | 落尾段 `56`/`14` → 假阳 | 歧义拒(None),不猜 ✓ |
| `p = .05` / `r = .32`(APA 无前导零) | 抓成 `5`/`32`(100×错) | `0.05` / `0.32` ✓ |
| `1 000 000` / `1 000,5`(ISO 空格千分位+逗号小数,含 NBSP) | 落尾组 `000` = 0 | `1000000` / `1000.5` ✓ |
| `3-5` / `2020-01-01` / `555-1234`(连字符) | 末数变**负**(`-5`…) | 连字符≠负号,取正 `5`… ✓ |
| `(3,-2)` / `roots 5,-3`(逗号后负号) | —— | 逗号后 `-` 仍是负号 `-2`/`-3` ✓ |
| `3/4` / `1/2^10`(分数,^ 先于 / 结合) | 取分母/裸指数 | 算成 `0.75` / `1/1024`(分母 0 拒) ✓ |
| `(-2)^0.5`(括号负底数小数指数 = 复数) | —— | 拒 None、不崩(不再 TypeError);无括号 `-2^0.5` = `-(2^0.5)` 合法实数 ✓ |
| `(-2)^2`(括号负底数乘方) | 落到裸指数 `2` | 算成 `4` ✓ |
| `-2^2`(无括号一元负号) | 当负底数算成 `4` | 按数学约定后结合 `-(2^2)` = `-4` ✓ |
| `2×10^6` / `1.5 x 10^-3`(系数科学计数) | 只取末位 `10^…` | 系数·10^指数 = `2000000` / `0.0015` ✓ |
| `O(n^2)` / `O(2^n)`(复杂度记号) | 指数/底数当答案 | 跳过,不当数 ✓ |
| `答案是 42,见第 7 章 / pages 12-13 / 12, 13 / page 3/4 / equation (7)`(引用,含区间/列表/分数/括号形,复数词) | 抓成引用数 → 判错/假阳 | 引用整段跳过,取 `42` ✓ |

`gold_answer` 也做同样归一(去千分位逗号)。取**最后一个**数字(最终答案通常在末尾)。坏 gold(非数字)
不崩、判 `(False, None)`。**已知局限**(宁拒不猜):日期不是标量(`2020-01-01` 取末段 1,判不对 gold=2020,
金标别出日期题);数值金标别用章节号当答案(引用跳过会误伤);无空格逗号列表 `5,3`/`(3,4)` 判为歧义拒 `None`(与欧式`3,14` 同一「宁拒不猜」取向,防假阳),标量数值题的金标别写成逗号元组。覆盖见 `tests/test_numeric_extraction.py`。

配套两条判分语义加固:**对冲不算弃答**——数值题答案里报了具体数字还挂「不确定」,按已作答记
(瞎编不能靠一句 not sure 洗成弃答);但弃答里的**顺带计数**豁免——「材料未涵盖,我查了全部 20 讲」的
`20 讲` 是材料规模陈述、不是报答案,弃答仍成立;豁免要**双条件**:数字后紧跟量词(讲/章/页/年/lectures…)**且**数字前有材料范围提示词(材料/全部/共/查了/checked/all…)——「not sure, maybe 5 years」这类**带单位的猜测**没有提示词,仍算已作答(对冲不豁免);**解析失败的数字尝试**(「可能是 3,14」「maybe 5/0」)同样算已作答——报了数≠没作答,判不出值只是判不出值。
**词法快路收紧**(`contains_gold`)——gold 是 ASCII 词时要求词边界(`microRAM` 不算含 `RAM`),gold 所在
**小句**内出现强否定(前后任一侧,`X is not the answer` / `X is wrong`)不走快路、交给 LLM 裁判;gold 出现
**多处**时逐处检查(「有人说不是 X,其实就是 X」第二处干净 → 判对)。快路返回 False 是安全方向(只是多花一次判分)。

## 2. 通用判分校准(`calibrate_matrix.py`)

取代 `calibrate.py` 硬编码的 algo/psyc——直接读 B4 `run_matrix` 的 `results_dir`(`answers.jsonl` +
`scores.jsonl`)配 `config.json` 的金标,任意课程可用。两步:

```bash
# 1) 抽分层样本(四层:可答判对/判错各半 + 越界弃答/未弃答,可答:越界≈2:1,避免 kappa 退化),写出**隐藏裁判判定**的待填表
python calibrate_matrix.py sample --results-dir <run_matrix 的 results_dir> --config <config.json> --n 30

# 2) 人工在 calibration/calibration_sheet.csv 的 human_correct 列填 1(对/可接受)/ 0(错),只看
#    question + gold_answer + reference_span 判 model_answer(越界题以「是否老实弃答」为准)。填完:
python calibrate_matrix.py kappa --results-dir <同上>
```

输出 Cohen's kappa(human, judge) + 原始一致率 + **人机分歧清单**(裁判最可能判错的地方)。判定对**盲**:
待填表里不含裁判判定(藏在 `.calibration_key.jsonl`),避免人被裁判带跑。分层样本用 `--seed` 可复现。

**样本构成**:数值题与词法快路(确定性判分、天然一致会灌高 kappa)排除;**越界探针保留**——它们的判定来自弃答检测器(关键词启发式,照样会错),人工按「是否老实弃答」判,这一致性正是校准对象之一。

完整性护栏(全部 fail-loud,宁可停也不出一张带偏的表):坏 JSONL 行报行号拒(缩样本会灌高 kappa);
`.run_meta.json` 指纹与 `--config` 不一致拒(防旧答案配新金标)、meta 损坏/缺指纹也拒、缺 meta 响亮警告;
answers/scores 同 key **重复行**拒(人标的答案可能不是裁判判的那条);判定行**配不上**答案/金标拒(账本失同步或 config 配错,静默丢会缩样本偏 kappa);`human_correct` 填了非 0/1
(yes/2 等)列出坏格拒,真空格才算「未填」;待填表单元格做 **Excel 公式注入防护**(`=`/`+`/`-`/`@` 开头
加 `'` 前缀——模型答案是不可信文本,不能直接进 Excel)。

> 先用 `--mock` 的 fixture 课程验证工具管线:`run_matrix.py --mock` 再 `calibrate_matrix.py sample`(mock
> 判定全同、分层不成,只验流程;真校准需真跑数据里判对判错都有)。

### 实测一次(2026-07-04,本方法学的首次真实校准)

用本节方法学对**真实 PSYC 110 / MIT 6.006 结果**(Sonnet 裁判)做了一轮人工校准:先过滤掉 29 条
配额错误"答案"(infra 残留,不是模型作答),再按 可答判对/判错 + 越界弃答/未弃答 **四层**分层
(`cmd_sample` 现即此方案:判对:判错各半、可答:越界≈2:1,n=24 时为 8/8/4/4)抽 24 条、
判分对标注者隐藏、逐条盲判——结果 **一致率 91.7%(22/24)、Cohen's kappa = 0.833**(高度一致 ≥0.8),
与早先 16 题抽查的 kappa = 0.875 互相印证。两条人机分歧**都是裁判偏严**(human=对 / judge=错,均为
OCEAN 题 closedbook 臂:`Extraversion` 拼写变体、`Openness to Experience` 展开式被判错)——方向上是**抽样迹象**(样本有限,
不构成对全部判分的证明):提示已发布正确率倾向保守而非虚高。这也当场验证了两条设计决策的价值:越界探针入样后,8 条 OOS 里人机 8/8 全一致
(弃答检测器这次没错判);infra 过滤若不做,24 条里会混进 1 条把"配额报错文本"当答案的假样本。
脱敏摘要(不含课程内容)入库在 `calibration/kappa_n24_summary.json`;盲表/人工判定属真实数据不入库。

## 3. 跨家族裁判(自我偏好)

裁判和被测生成器**同一模型家族**(都 Claude:opus/sonnet/haiku)时有**自我偏好**嫌疑——模型倾向
认可自家风格的答案。`calibrate_matrix sample` 会据 `summary.json` 的 `judge_model` 与作答行的生成器家族
比对,重叠即警告。

**局限(诚实说明)**:`run_matrix` 的判分路径 `_real_ask_judge` 只走 `gen.run_claude(prompt, judge_model)`
——即 `claude -p --model <judge_model>`,**只能用 Claude 家族**当裁判;它**不读** `openai_api_base`,
现阶段**没有**把 Gemini / GPT / DeepSeek 接进来的通路。所以要跨家族裁判,只能**在 run_matrix 之外**另跑
一个非 Claude 裁判(把 `scores.jsonl` 的判定用别的模型重算后回填),或先在报告里注明「裁判与被测同属
Claude 家族」这一局限。跨家族裁判接入属**待补**(future work),别照着 `openai_api_base` 去配——配了没用。

## 4. near-miss 越界探针(出题建议)

「越界弃答」这项容易被**太明显**的越界题刷高分——如问「亚马逊雨林降水量」这种和课程八竿子打不着的题,
模型当然弃答。要真考验抗幻觉,越界探针该出 **near-miss**:**话题贴着材料、但材料里没有确切答案**的题。
例:材料讲了 Gearbox 调度器的时间片(50ms),就出「Gearbox 调度器的默认优先级提升间隔是多少?」——
调度器在讲义里、但这个具体参数没有,正确行为仍是老实弃答。这类题才分得出「真读了材料并知道边界」和
「看着相关就编」。出金标时(`items.jsonl`,`answerable=false`)**优先出这种贴边的越界题**,而非明显跑题的。

---

诚实小结:本层是**校准与判分质量**的加固,不改变任何已发布数字;它让「裁判可信度」有据可查(kappa),
并把数值判分、跨家族偏好、越界探针难度这三个此前的软肋补上。纯 Python 标准库、零依赖。
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall