Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/matrix_pipeline.en.md

View on GitHub
# Matrix Benchmark Pipeline

English · [中文](matrix_pipeline.md)

This document describes, precisely, the pipeline "answers/scores → matrix `summary.json` → report", and **the piece T3 filled in**: a committed, tested aggregation/conversion script that makes the pipeline's **mechanism** reproducible from a clean checkout using fixture data.

> Important boundary: **T3 makes the pipeline mechanism reproducible with fixture data; it does not make the "published full MIT/PSYC benchmark" rerunnable from a clean checkout**. The full matrix still depends on private/local intermediate artifacts and paid model calls (see §2, §5).

## 1. What exists today

| Stage | Script | Notes |
| --- | --- | --- |
| Generate | `benchmark/gen.py` | An **incremental gap-filling** helper: it fills missing cells from an existing `answers.jsonl`. It is **not** a one-command full-matrix generator. |
| Judge | `benchmark/judge.py` / `benchmark/rejudge.py` | Exact numeric comparison + lexical exact-match + abstention; optional LLM re-judging (cached). `rejudge.py` contains the per-cell `aggregate()` logic, but it is coupled to judging and to reading `results/`. **Since T3.1**, `rejudge.py` supports `--scores-out` (**`--answers-out` must be given too**) to export, at zero cost, score/answer rows that `aggregate_matrix.py` can read (answer rows carry `status`/`cost_usd`). This is the bridge between judging and aggregation. |
| **Aggregate** | **`benchmark/aggregate_matrix.py` (new in T3)** | Reads **explicit** answer/score rows → a matrix summary compatible with `summary.json`. Pure standard library, deterministic, no network/LLM. |
| Render | `benchmark/report_matrix.py` | summary → bilingual Chinese/English `report.html` + chart SVGs. Supports `--summary` / `--out-dir` since T3. |

## 2. What a clean checkout actually contains

- **Published matrix artifacts** (`results/matrix/summary.json`, `report.html`, chart SVGs) are committed to the repository.
- **Private/intermediate answer artifacts** (real model answer logs, private gold labels, caches from paid runs) are **not** committed.
- So rerunning the "full MIT 6.006 / Yale PSYC 110 matrix" from scratch **still requires** private/local inputs + paid model runs. T3 does not change that, and it **does not change any published number**.

## 3. What T3 added

1. **A committed aggregation/conversion script**, `aggregate_matrix.py`: it aggregates explicit answer/score rows into a summary (it is not tangled up with judging and reading `results/` the way `rejudge.py` is, and it **never** silently reads `results/matrix/summary.json`).
2. **A fixture-level reproducible pipeline**: `benchmark/tests/fixtures/matrix_pipeline/` (self-authored, tiny, not copyrighted) runs aggregation + rendering end to end.
3. **An explicit `report_matrix.py --summary <file> --out-dir <dir>`**: the renderer is no longer forced to render only the committed summary.

### Input/output schema

- **answers.jsonl** (the full item set + each answer attempt): required `course, model, arm, item_id`; optional `answerable` (`bool`/`0`/`1`. It may instead go on the matching score row, which is this repository's `gen.py→judge` path; but **at least** one of the answer or the score must provide it, or the script fails loudly. **If both sides provide it and they disagree**, the script fails loudly for that `(course,model,arm,item_id)` and does not silently favor either side), `status` (`"ok"`/`"infra_error"`, default `ok`), `cost_usd`. Two in-repo row aliases, `id`→`item_id` and `cost`→`cost_usd`, are also accepted.
- **scores.jsonl** (judging): required `course, model, arm, item_id`; optional `correct`, `hallucinated`, `abstained`, `judge_error` (`bool` or `0`/`1`; `judge.py` actually writes integers), `faithfulness` (numeric, must be within `[0,1]`), `answerable`, `scored_by`.
- **summary (output)**: `matrix` (the primary course's `model|arm` cells; this is the block the renderer reads), `course_matrix` (**general**: cells for every course), `cost_per_q` (per-question cost, course → arm), `psyc` (a renderer-compatible block for `--secondary-course`), `models/arms/courses/n_items/total_cost_usd/judge_model`.

### Honest counting rules (aligned with `rejudge.aggregate()`)

- An `infra_error` (rate limit/context error) is **not** a model answer → it is **excluded** from the accuracy denominator but counted in `n_infra_error`. It is **never silently dropped**.
- An answered answerable item with **no matching score** → recorded as `judge_error` and counted as **wrong** (a trustworthy lower bound). A missing score **never** raises accuracy.
- `correct`/`hallucination` are rates over completed answerable items; `faithfulness` is averaged over items that were actually judged; `abstention_oos` is computed over completed out-of-scope items. Failed/errored/missing cells are reported as they are; failures are **not** swallowed in any way that inflates the metrics.

## 4. Fixture commands (runnable on a clean checkout)

```bash
python benchmark/aggregate_matrix.py \
  --answers benchmark/tests/fixtures/matrix_pipeline/answers.jsonl \
  --scores  benchmark/tests/fixtures/matrix_pipeline/scores.jsonl \
  --primary-course courseA --secondary-course courseB \
  --out /tmp/examprep-summary.json

python benchmark/report_matrix.py \
  --summary /tmp/examprep-summary.json \
  --out-dir /tmp/examprep-report
```

## 5. Expected path for a real run (needs private inputs + paid calls; **not run in this PR**)

```text
gen.py (fill in/generate answers, real model, paid)
   → judge.py / rejudge.py (judge/re-judge, optional LLM)
   → rejudge.py --deterministic --scores-out (zero-cost score-row export: the bridge between judging and aggregation)
   → aggregate_matrix.py (aggregate, zero cost)
   → report_matrix.py --summary <summary.json> --out-dir <dir> (render)
```

Explicit commands you can run one at a time (on a machine that **has the real artifacts**; the export, aggregate and render steps themselves cost nothing and do not call an LLM):

```bash
# 1) Zero-cost export of score rows (+ answer rows aligned strictly by (course,model,arm,item_id)).
#    Note: rejudge.py still reads the private/local results artifacts and gold labels. On a clean checkout these files are missing and it raises FileNotFoundError;
#    run this step on a machine with the real artifacts. Deterministic mode calls no LLM. The export writes only to the explicit paths given by
#    --scores-out/--answers-out; rejudge itself still writes its (gitignored) summary_corrected.json/progress files to
#    results/matrix/ as before (existing behavior), but it NEVER touches the published results/matrix/summary.json or report.html.
python benchmark/rejudge.py --deterministic \
  --scores-out  /tmp/scores.jsonl \
  --answers-out /tmp/answers.jsonl

# 2) Aggregate into a summary (pure standard library, zero cost, deterministic)
python benchmark/aggregate_matrix.py \
  --answers /tmp/answers.jsonl --scores /tmp/scores.jsonl \
  --primary-course algo --secondary-course psyc \
  --out /tmp/summary.json

# 3) Render into a CUSTOM directory (always pass --out-dir)
python benchmark/report_matrix.py \
  --summary /tmp/summary.json --out-dir /tmp/report
```

A few points need to be stated plainly:

- **The score-row export is the bridge**: `rejudge.py --scores-out` (**`--answers-out` must be given too**) normalizes each item's score into the fields `aggregate_matrix.py` needs (`answerable` is carried over from the gold labels; `scored_by` keeps the judge's own label, lexical/llm/judge_error/infra_error), and it also carries over **the answer rows' `status` (infra_error) and `cost_usd`**. Without them the aggregator would count infra failures as answerable items and compute the cost as `$0`. **It exports only matrix/psyc cells** (not the `conv_*` convergence rounds, to avoid `(course,model,arm,item_id)` key collisions; a duplicate key **fails loudly** instead of being silently dropped). It is off by default, zero cost, calls no LLM, and the export writes only to explicit paths.
- **`summary_corrected.json` cannot be rendered directly**: by default `rejudge.py` still writes its original `summary_corrected.json` (a nested `algo`/`psyc` structure). That shape is **different** from the top-level `matrix`/`n_items` that `report_matrix.py` reads, so you **cannot** feed it straight to the renderer with `--summary summary_corrected.json`. Go through `--scores-out → aggregate_matrix.py` to get a summary the renderer can read.
- **A custom summary requires `--out-dir`**: rendering a **custom** `--summary` without `--out-dir` (that is, into the default `results/matrix/`) is **refused** (exit code 2) so that it cannot overwrite the published `results/matrix/report.html`. Always point it somewhere else.

## 6. What this PR does not do

- It does **not** rerun the real MIT/PSYC benchmark, and it does **not** change any published result number or chart.
- It does **not** commit private answer artifacts, private gold labels, paid-run artifacts, or any copyrighted course text/page images.
- It does **not** redesign the report's visual style, and it does **not** change the skill's runtime behavior, `scripts/ingest.py`, or the P0A/P0B/P0D material builders.
- It does **not** add dependencies, GitHub Actions, or any network/LLM/API calls.

> Wording standard: **"the pipeline mechanism is reproducible with fixture data"**, not "the full published MIT/PSYC benchmark can now be reproduced from a clean checkout". The latter still needs private/intermediate artifacts and paid runs. The P0A/P0B material builders are **unrelated** to this benchmark aggregation path and should not be confused with it.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall