Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/running-real-runs.md

View on GitHub
# 真·付费 LLM 跑:操作手册(opt-in)

[English](running-real-runs.en.md) · 中文

> 面向仓库拥有者(有 Claude Code 订阅、**无独立 API key**)。所有真跑都通过 shell `claude -p` 走你已登录的订阅,不需要 provider API key。
> **先用 `--mock` 免费验证管线,再加 `--real`。** 真跑消耗你的订阅配额;所有 runner 都 resumable + 配额感知,Ctrl-C 安全、`--real` 重跑自动续。
> CI 里这些真跑**永远不开**——全靠显式 flag / env 门控。真实材料、金标、结果都 `.gitignore` 挡住,不进公开仓库。

前提(本地已就绪,均 gitignored):
- 课程材料 `materials/algorithms_mit6006/`(6.006)、`materials/psych_yale_psyc110/`(PSYC 110)。
- 金标 `items/items_algo_full.jsonl`(69 题)、`items/items_psyc_full.jsonl`(50 题,含 10 道越界探针 `answerable:false`)。
- 各臂工作区 `skill_workspace/{mit6006_full,rawfiles_algo,psyc110_full,rawfiles_psyc}`。
- 已 `claude` 登录(`claude -p "hi"` 能出话)。

---

## 主线:三臂 × 三模型矩阵(`run_matrix.py`)

三臂 = **closedbook**(不给材料,暴露先验知识底噪)/ **rawfiles**(agent 在原始课件目录里 Read/Glob/Grep)/ **skill**(agent 在 skill 工作区,惰性加载 wiki)。三模型 = opus / sonnet / haiku。判分内联(生成完当场判),不用另开判分步骤。

### 第 0 步 · 免费验证管线(不打 claude)
```bash
cd benchmark
python run_matrix.py --config config.matrix.example.json --mock --limit 6
```
应打印 `summary: …/results/matrix_run/summary.json(mock 占位摘要,未测量正确率)`。跑通即证明 config 里的课程/工作区/金标路径都接对了。**跑完删掉占位目录**再真跑(避免 mock 占位被当成已完成):
```bash
rm -rf results/matrix_run
```

### 第 1 步 · 建你的真跑 config
复制模板(模板已按真实两门课填好路径):
```bash
cp config.matrix.example.json config.json
```
`config.json` 是 gitignored 的个人配置。**建议改一处**:把 `"judge_model"` 从 `"haiku"` 改成 `"sonnet"`——Sonnet 是人工 kappa 校准过的裁判(human vs Sonnet kappa=0.833,见 [`judge-calibration.md`](judge-calibration.md)),haiku 判分更弱。
> ⚠️ 自判偏好提示:裁判模型也在生成模型集里时,同名那一臂(sonnet 判 sonnet)存在自我偏好嫌疑。已发布方法学用的就是 Sonnet 裁判并如实标注此口径;若要彻底规避,需换一个**不在生成集里**的裁判(如 Codex/GPT),但那需要另一套接入。默认沿用 Sonnet 裁判 + 明示口径。

`results_dir` 默认 `results/matrix_run`;真跑建议单独目录(别覆盖已发布的 `results/matrix/`):把 config 里 `results_dir` 设为如 `results/matrix_real_2026xxxx`。

### 第 2 步 · 真跑(消耗配额,resumable)
```bash
python run_matrix.py --config config.json --real
```
- 全矩阵 = 2 课 × 3 模型 × 3 臂 × (69+50) 题,量很大;**先小样验证**:`--real --limit 20` 或先把 config 的 `models` 砍成 `["haiku"]`、`arms` 砍成 `["closedbook","skill"]` 跑一轮看通。
- 撞订阅 5 小时配额上限会停并提示「配额未恢复,稍后再跑 --real 续」——过几小时再跑同一条命令自动续,已判分的 `(课,模型,臂,题)` 跳过。
- 产物写在 `results_dir/`:`answers.jsonl`(含 `cost_usd`)、`scores.jsonl`、`.run_meta.json`(config 指纹,防 mock/real 混目录)、`summary.json`(全跑完且无 infra_error 才落)。

### 第 3 步 · 聚合 + 报告
`run_matrix.py` 全跑完会自动聚合出 `summary.json`。若中途续跑或想单独重生成:
```bash
# 聚合(纯 stdlib,不打 claude)
python aggregate_matrix.py --answers results/matrix_real_xxx/answers.jsonl \
  --scores results/matrix_real_xxx/scores.jsonl --out results/matrix_real_xxx/summary.json \
  --primary-course algo --secondary-course psyc --judge-model sonnet
# 渲染中英双语 HTML + SVG(自定义 --summary 时必须给 --out-dir,否则退出码 2 防覆盖已发布报告)
python report_matrix.py --summary results/matrix_real_xxx/summary.json --out-dir results/matrix_real_xxx
# 打开 results/matrix_real_xxx/report.html
```
重判(只对已缓存答案,不重新生成):`python rejudge.py --deterministic --course both --scores-out … --answers-out …`(`--judge-model` 默认 sonnet;`--llm` 才对判不定项调 claude)。

---

## 可选 A · 行为冒烟真跑(`behavior_smoke --llm`,B2)
确定性 mock 免费,真 agent 单轮冒烟 opt-in:
```bash
python behavior_smoke/run_behavior_smoke.py --mock          # 免费,确定性探测器
RUN_SKILL_BEHAVIOR_LLM=1 python behavior_smoke/run_behavior_smoke.py --llm   # 真跑(默认 claude -p {prompt})
# 或不设 env、显式给命令:
python behavior_smoke/run_behavior_smoke.py --llm --agent-cmd "claude -p {prompt}"
```
每场景在**含 skill 契约的一次性沙箱**里跑一轮,套用与 `--mock` 相同的确定性探测器;transcript 默认写到仓库外系统临时目录(含答案键,别落工作树)。退出码:0 全过 / 1 有场景不过 / 2 门控或参数错 / 3 中途超时或超输出上限被杀。

## 可选 B · 长会话漂移真跑(`drift/run_live_smoke.py`)
需**同时**给 `--agent-cmd` 和 `RUN_SKILL_DRIFT_LLM=1`:
```bash
RUN_SKILL_DRIFT_LLM=1 python drift/run_live_smoke.py \
  --agent-cmd "claude -p {prompt}" --out-dir /tmp/live_smoke
```
`--turns` 默认自带回合脚本;`--out-dir` 必须显式(不写任何 `results/`)。驱动→记录→T4 判分一条命令。

## 可选 C · 多轮收敛(`rounds.py` + `mit6006_r{1,2,3}` 残缺 wiki)
收敛协议是**人工多轮**:作答→判分→自检缺哪块材料→补 wiki(r1=7章→r2=14章→r3=20章)→下一轮重答。每轮指标喂 `rounds.render_convergence(rounds=[…], out_dir=…, mock=False)` 出 `convergence.html`;faithfulness 连续两轮 Δ<2% 判收敛。

---

## 配额 / 成本 / 诚实口径
- **配额**:订阅有 5 小时滚动上限;大 workflow/矩阵会撞。分批跑、`--limit` 小样先行、撞上限就等恢复续跑。
- **成本**:`answers.jsonl` 每行带 `cost_usd`,`summary.json` 汇总 `total_cost_usd`。
- **诚实**:`infra_error`(配额/超时/判分失败)答案**排除出正确率分母**、单独计 `n_infra_error`,绝不当成「答错/幻觉」——这是之前踩过的坑(120 条配额错误曾把 haiku|material 假压到 2%)。越界探针(`answerable:false`)判「弃答=对 / 硬答=幻觉」。
- **判分可信**:数值题确定性判;事实题先 `contains_gold` 词边界快路径,判不定才调裁判(claim 级蕴含 vs supporting_span)。人工 kappa=0.833/0.875(保守下界,非虚高)。

## 更难金标(压低闭卷正确率)
现有金标里不少题模型靠先验就能答(PSYC 闭卷 ~60%、6.006 部分同理),grounding 考验不够狠。`items/items_psyc_hard.jsonl`(**54 题,gitignored**)是一批**只能从转录里取答**的高 grounding 题——都是 Bloom 教授本人举的具体例子/私人轶事/点名的冷门研究/具体数字(如他小时候的电话号码 514-688-9057、小脑约 300 亿神经元、Pratfall 实验 92%、他儿子说"要娶一头驴和一大袋花生"),**常识答不出、只有看过该讲才知道**。每题的 `supporting_span` 都是从转录里**逐字复制**并经脚本校验确属该讲文件(1 条非逐字的已剔除)。全部标 `answer_type:"factual"`(数字答案也含词/单位/区间,走事实判分而非确定性数值比对)。

用法:真跑时把 config 的 psyc `items` 指向它(或与 `items_psyc_full.jsonl` 合并),**闭卷该明显掉、skill/rawfiles 该守住**——那才坐实 grounding 有用。生成方法:5 个 subagent 分读 20 讲转录挖题 + 逐字 span 校验 + 去重(脚本一次性,非仓库常驻)。其**难度只有真跑才能验证**(离线无法确认模型先验),跑完对比 closedbook vs skill/rawfiles 的正确率差。
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall