Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/running-real-runs.en.md

View on GitHub
# Real Paid LLM Runs: Operator Guide (opt-in)

English · [中文](running-real-runs.md)

> For the repository owner (who has a Claude Code subscription and **no separate API key**). Every real run goes through the shell `claude -p` on your logged-in subscription; no provider API key is needed.
> **Validate the pipeline for free with `--mock` first, then add `--real`.** Real runs consume your subscription quota. Every runner is resumable and quota-aware, safe to Ctrl-C, and rerunning with `--real` resumes automatically.
> These real runs are **never turned on** in CI; they are gated entirely by explicit flags/env vars. Real materials, gold labels and results are all blocked by `.gitignore` and never enter the public repository.

Prerequisites (already in place locally, all gitignored):
- Course materials `materials/algorithms_mit6006/` (6.006) and `materials/psych_yale_psyc110/` (PSYC 110).
- Gold labels `items/items_algo_full.jsonl` (69 items) and `items/items_psyc_full.jsonl` (50 items, including 10 out-of-scope probes with `answerable:false`).
- Per-arm workspaces `skill_workspace/{mit6006_full,rawfiles_algo,psyc110_full,rawfiles_psyc}`.
- Logged in to `claude` (`claude -p "hi"` produces a reply).

---

## Main line: three arms × three models matrix (`run_matrix.py`)

The three arms = **closedbook** (no materials; exposes the prior-knowledge noise floor) / **rawfiles** (the agent uses Read/Glob/Grep in the raw course-file directory) / **skill** (the agent works in the skill workspace and lazily loads the wiki). The three models = opus / sonnet / haiku. Judging is inline (each answer is judged right after it is generated), so there is no separate judging step.

### Step 0 · Validate the pipeline for free (no claude calls)
```bash
cd benchmark
python run_matrix.py --config config.matrix.example.json --mock --limit 6
```
It should print `summary: …/results/matrix_run/summary.json(mock 占位摘要,未测量正确率)` ("mock placeholder summary; accuracy not measured"). If it runs through, the course/workspace/gold-label paths in the config are wired correctly. **Delete the placeholder directory afterwards**, before the real run (so the mock placeholders are not mistaken for completed work):
```bash
rm -rf results/matrix_run
```

### Step 1 · Build your real-run config
Copy the template (it already has the paths filled in for the two real courses):
```bash
cp config.matrix.example.json config.json
```
`config.json` is a gitignored personal config. **One change is recommended**: switch `"judge_model"` from `"haiku"` to `"sonnet"`. Sonnet is the judge that was calibrated against human kappa (human vs Sonnet kappa=0.833, see [`judge-calibration.en.md`](judge-calibration.en.md)); haiku is a weaker judge.
> ⚠️ Self-preference warning: when the judge model is also one of the generating models, the matching cells (sonnet judging sonnet) are open to self-preference bias. The published methodology uses exactly this Sonnet judge and states this basis openly. To rule it out entirely you would need a judge that is **not in the generating set** (such as Codex/GPT), but that requires a separate integration. The default is to keep the Sonnet judge and state the basis explicitly.

`results_dir` defaults to `results/matrix_run`. For a real run, use a separate directory (do not overwrite the published `results/matrix/`): set `results_dir` in the config to something like `results/matrix_real_2026xxxx`.

### Step 2 · Real run (consumes quota, resumable)
```bash
python run_matrix.py --config config.json --real
```
- The full matrix = 2 courses × 3 models × 3 arms × (69+50) items, which is a lot. **Validate on a small sample first**: use `--real --limit 20`, or first cut the config's `models` down to `["haiku"]` and `arms` to `["closedbook","skill"]` and run one round to confirm it works.
- Hitting the subscription's 5-hour quota cap stops the run with the message 「配额未恢复,稍后再跑 --real 续」 ("quota not restored; rerun --real later to resume"). Run the same command again a few hours later and it resumes automatically, skipping `(course, model, arm, item)` combinations that are already scored.
- Artifacts are written to `results_dir/`: `answers.jsonl` (includes `cost_usd`), `scores.jsonl`, `.run_meta.json` (a config fingerprint that keeps mock and real runs from sharing a directory), and `summary.json` (written only when the whole run finishes with no infra_error).

### Step 3 · Aggregate + report
`run_matrix.py` aggregates `summary.json` automatically once the whole run finishes. If you resumed partway through or want to regenerate it separately:
```bash
# Aggregate (pure stdlib, no claude calls)
python aggregate_matrix.py --answers results/matrix_real_xxx/answers.jsonl \
  --scores results/matrix_real_xxx/scores.jsonl --out results/matrix_real_xxx/summary.json \
  --primary-course algo --secondary-course psyc --judge-model sonnet
# Render bilingual Chinese/English HTML + SVG (a custom --summary requires --out-dir; otherwise exit code 2 prevents overwriting the published report)
python report_matrix.py --summary results/matrix_real_xxx/summary.json --out-dir results/matrix_real_xxx
# Open results/matrix_real_xxx/report.html
```
Re-judging (cached answers only, no regeneration): `python rejudge.py --deterministic --course both --scores-out … --answers-out …` (`--judge-model` defaults to sonnet; only `--llm` calls claude for undecided items).

---

## Optional A · Real behavior-smoke run (`behavior_smoke --llm`, B2)
The deterministic mock is free; the single-turn real-agent smoke is opt-in:
```bash
python behavior_smoke/run_behavior_smoke.py --mock          # free, deterministic detectors
RUN_SKILL_BEHAVIOR_LLM=1 python behavior_smoke/run_behavior_smoke.py --llm   # real run (default claude -p {prompt})
# Or, without setting the env var, pass the command explicitly:
python behavior_smoke/run_behavior_smoke.py --llm --agent-cmd "claude -p {prompt}"
```
Each scenario runs one turn in a **throwaway sandbox that contains the skill contract** and applies the same deterministic detectors as `--mock`. Transcripts go by default to the system temp directory outside the repository (they contain answer keys, so keep them out of the working tree). Exit codes: 0 all pass / 1 some scenario failed / 2 gating or argument error / 3 killed midway for a timeout or for exceeding the output limit.

## Optional B · Real long-session drift run (`drift/run_live_smoke.py`)
Requires **both** `--agent-cmd` and `RUN_SKILL_DRIFT_LLM=1`:
```bash
RUN_SKILL_DRIFT_LLM=1 python drift/run_live_smoke.py \
  --agent-cmd "claude -p {prompt}" --out-dir /tmp/live_smoke
```
`--turns` defaults to the built-in turn script; `--out-dir` must be given explicitly (nothing is written to any `results/`). Drive → record → T4 scoring in one command.

## Optional C · Multi-round convergence (`rounds.py` + the incomplete `mit6006_r{1,2,3}` wikis)
The convergence protocol is **manual and multi-round**: answer → judge → self-check which materials are missing → add to the wiki (r1=7 chapters → r2=14 chapters → r3=20 chapters) → answer again in the next round. Each round's metrics are fed to `rounds.render_convergence(rounds=[…], out_dir=…, mock=False)` to produce `convergence.html`; faithfulness counts as converged when Δ<2% for two consecutive rounds.

---

## Quota / cost / honest reporting
- **Quota**: the subscription has a rolling 5-hour cap, and large workflows/matrices will hit it. Run in batches, start with a small `--limit` sample, and when you hit the cap, wait for it to recover and resume.
- **Cost**: every row of `answers.jsonl` carries `cost_usd`, and `summary.json` totals it in `total_cost_usd`.
- **Honesty**: `infra_error` answers (quota/timeout/judging failure) are **excluded from the accuracy denominator** and counted separately in `n_infra_error`. They are never treated as "wrong/hallucinated". We fell into this trap before (120 quota errors once dragged haiku|material down to a false 2%). Out-of-scope probes (`answerable:false`) are scored as "abstain = correct / forced answer = hallucination".
- **Trustworthy judging**: numeric items are judged deterministically. Factual items first take the `contains_gold` word-boundary fast path and call the judge only when that is inconclusive (claim-level entailment vs supporting_span). Human kappa=0.833/0.875 (a conservative lower bound, not inflated).

## Harder gold labels (to push closed-book accuracy down)
Models can answer many items in the existing gold set from prior knowledge (PSYC closed-book ~60%, and parts of 6.006 likewise), so the set does not test grounding hard enough. `items/items_psyc_hard.jsonl` (**54 items, gitignored**) is a batch of high-grounding items that can **only be answered from the transcripts**. They are all specific examples, personal anecdotes, obscure studies he names, or specific numbers from Professor Bloom himself (such as his childhood phone number 514-688-9057, the cerebellum's roughly 30 billion neurons, 92% in the Pratfall experiment, and his son saying he wanted "to marry a donkey and a big bag of peanuts"). **Common knowledge cannot answer them; only someone who watched that lecture would know.** Each item's `supporting_span` is **copied verbatim** from the transcript and checked by a script to belong to that lecture's file (1 non-verbatim item was removed). All are labeled `answer_type:"factual"` (numeric answers also include words/units/ranges, so they go through factual judging rather than deterministic numeric comparison).

Usage: for a real run, point the config's psyc `items` at it (or merge it with `items_psyc_full.jsonl`). **Closed-book should drop noticeably while skill/rawfiles hold up**; only that would confirm that grounding helps. How it was built: 5 subagents each read part of the 20 lecture transcripts to mine items, followed by verbatim span verification and de-duplication (one-off scripts, not kept in the repository). Its **difficulty can only be verified by a real run** (there is no offline way to confirm what the models already know); after the run, compare the accuracy gap between closedbook and skill/rawfiles.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall