Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/aggregate_matrix.py

View on GitHub
# -*- coding: utf-8 -*-
"""Aggregate EXPLICIT answer + score rows into a summary.json-compatible matrix summary that
benchmark/report_matrix.py can render. Pure stdlib · deterministic · no network / LLM / API keys / deps.

This is the committed, tested aggregation step the matrix pipeline was missing. It takes explicit
file inputs (never silently reads results/matrix/summary.json) and emits a fresh summary:

    generated answers / judge scores
            ↓  aggregate_matrix.py  (this script)
    summary.json-compatible matrix summary
            ↓  report_matrix.py --summary <summary.json> --out-dir <dir>

Inputs (JSONL, one JSON object per line). `id`→item_id and `cost`→cost_usd are accepted (this repo's
gen.py / judge.py row shapes), and boolean flags accept JSON booleans OR 0/1 (judge.py emits integers):
  --answers : the ITEM UNIVERSE + each answer attempt. Required: course, model, arm, item_id.
              Optional: answerable (bool/0/1 — may instead live on the score row, the gen.py→judge
              path; if BOTH the answer and score rows carry it they must AGREE, a conflict fails loud),
              status ("ok" | "infra_error", default "ok"), cost_usd (number, default 0).
  --scores  : the JUDGMENTS. Required: course, model, arm, item_id. Optional: correct, hallucinated,
              abstained, judge_error (bool/0/1), faithfulness (number in [0,1]), answerable, scored_by.

Honesty rules (mirrors benchmark/rejudge.aggregate() so fixture and real runs agree):
  - infra_error answers are EXCLUDED from correctness denominators (they are not model answers) but
    counted in n_infra_error — never silently dropped.
  - a completed answerable item with NO matching score row is counted as judge_error (NOT correct):
    a missing judgment never inflates correctness.
  - correctness/hallucination are rates over completed answerable items; faithfulness averages over
    items that were actually judged (not judge_error). abstention_oos is over completed OOS items.

Output (--out): models / arms / courses / n_items / total_cost_usd / judge_model, a `matrix` block for
the --primary-course (the model|arm cells report_matrix.py reads), a GENERIC `course_matrix`
(every course's cells), `cost_per_q` (course→arm mean per-item cost), and — when --secondary-course is
given — a renderer-compatible `psyc` block (psyc|model|arm keys) for that course.
"""
import argparse
import json
import os
import sys

REQ_ANSWER = ("course", "model", "arm", "item_id")   # `answerable` may instead live on the score row
REQ_SCORE = ("course", "model", "arm", "item_id")


def _die(msg):
    sys.stderr.write("aggregate_matrix: " + msg + "\n")
    raise SystemExit(2)


def _load_jsonl(path, required, label, alias=None):
    if not os.path.isfile(path):
        _die("找不到 %s 文件: %s" % (label, path))
    rows = []
    with open(path, "r", encoding="utf-8") as f:
        for ln, line in enumerate(f, 1):
            line = line.strip()
            if not line:
                continue
            try:
                d = json.loads(line)
            except ValueError as e:
                _die("%s 第 %d 行不是合法 JSON: %s" % (label, ln, e))
            if not isinstance(d, dict):
                _die("%s 第 %d 行必须是 JSON 对象" % (label, ln))
            if alias:
                alias(d)   # normalize repo-native aliases (id→item_id, cost→cost_usd) before the field check
            missing = [k for k in required if k not in d]
            if missing:
                _die("%s 第 %d 行缺必需字段 %s" % (label, ln, missing))
            rows.append(d)
    return rows


def _key(d):
    return (d["course"], d["model"], d["arm"], str(d["item_id"]))


def _validate_score(s):
    # accept JSON booleans OR 0/1 (this repo's judge.py emits integer flags); reject string-encoded
    # booleans etc. — "false" is truthy and would silently corrupt the rates. rate() treats 0/1 right.
    for b in ("correct", "hallucinated", "abstained", "judge_error"):
        v = s.get(b)
        if v is not None and not (isinstance(v, bool) or (isinstance(v, int) and v in (0, 1))):
            _die("scores %s 的 %s 必须是布尔值或 0/1(或省略/null),不能是 %r" % (_key(s), b, v))
    f = s.get("faithfulness")
    if f is not None:
        if isinstance(f, bool) or not isinstance(f, (int, float)):
            _die("scores %s 的 faithfulness 必须是数值(或省略/null),不能是 %r" % (_key(s), f))
        if not (0 <= f <= 1):
            _die("scores %s 的 faithfulness 必须在 [0,1] 内,当前 %r" % (_key(s), f))


def _as_bool(v):
    """Coerce a flag to bool — accepts JSON booleans and 0/1 (judge.py emits integers); None passes through."""
    if v is None or isinstance(v, bool):
        return v
    if isinstance(v, int) and v in (0, 1):
        return bool(v)
    return None  # caller treats this as 'unusable'


def _alias_answer(d):
    # accept this repo's gen.py row shape: `id` → item_id, `cost` → cost_usd
    if "item_id" not in d and "id" in d:
        d["item_id"] = d["id"]
    if "cost_usd" not in d and "cost" in d:
        d["cost_usd"] = d["cost"]


def _alias_score(d):
    if "item_id" not in d and "id" in d:   # judge.py / rejudge.py score dicts carry `id`
        d["item_id"] = d["id"]


def _cell(items):
    """Compute one matrix cell from merged per-item dicts. Mirrors benchmark/rejudge.aggregate()
    (+ a cost_usd total). `items` carry: answerable, infra_error, judge_error, correct, faithfulness,
    hallucinated, abstained, scored_by, cost_usd."""
    ans = [s for s in items if s["answerable"] and not s["infra_error"]]
    oos = [s for s in items if not s["answerable"] and not s["infra_error"]]
    decided = [s for s in ans if not s["judge_error"] and s.get("faithfulness") is not None]

    def rate(xs, key):
        xs = [x for x in xs if x.get(key) is not None]
        return round(sum(1 for x in xs if x.get(key)) / len(xs), 4) if xs else None

    return {
        "n": len(items), "n_answerable": len(ans), "n_oos": len(oos),
        "correct": rate(ans, "correct"),
        "faithfulness": round(sum(s["faithfulness"] for s in decided) / len(decided), 4) if decided else None,
        "hallucination": rate(ans, "hallucinated"),
        "abstention_oos": rate(oos, "abstained"),
        "n_judge_error": sum(1 for s in ans if s["judge_error"]),
        "n_lexical": sum(1 for s in ans if s.get("scored_by") == "lexical"),
        "n_infra_error": sum(1 for s in items if s["infra_error"]),
        "cost_usd": round(sum((s.get("cost_usd") or 0) for s in items), 6),
    }


def merge_rows(answers, scores):
    """Join answers (item universe) with scores (judgments) by (course, model, arm, item_id).
    Returns a list of merged per-item dicts the cell aggregator consumes."""
    score_by = {}
    for s in scores:
        k = _key(s)
        if k in score_by:
            _die("scores 中出现重复 (course,model,arm,item_id): %s" % (k,))
        _validate_score(s)
        score_by[k] = s
    seen = set()
    merged = []
    for a in answers:
        k = _key(a)
        if k in seen:
            _die("answers 中出现重复 (course,model,arm,item_id): %s" % (k,))
        seen.add(k)
        cost = a.get("cost_usd", 0) or 0
        if isinstance(cost, bool) or not isinstance(cost, (int, float)):
            _die("answers %s 的 cost_usd 必须是数值,当前 %r" % (k, cost))
        status = a.get("status", "ok")
        if status not in ("ok", "infra_error"):
            _die("answers %s 的 status 必须是 'ok' 或 'infra_error',当前 %r" % (k, status))
        infra = status == "infra_error"
        s = score_by.get(k)
        # `answerable` may live on the answer row OR (the documented gen.py→judge path) on the score
        # row. A side "provides" it only when the key exists AND is non-null. When BOTH sides provide
        # it we REQUIRE agreement and fail loudly on a conflict — we never silently prefer one side.
        a_has = "answerable" in a and a["answerable"] is not None
        s_has = s is not None and "answerable" in s and s["answerable"] is not None
        if not a_has and not s_has:
            _die("answers/scores %s 都没有 answerable(无法区分可答题/越界探针题)" % (k,))
        a_val = s_val = None
        if a_has:
            a_val = _as_bool(a["answerable"])
            if a_val is None:
                _die("answers %s 的 answerable 必须是布尔或 0/1,当前 %r" % (k, a["answerable"]))
        if s_has:
            s_val = _as_bool(s["answerable"])
            if s_val is None:
                _die("scores %s 的 answerable 必须是布尔或 0/1,当前 %r" % (k, s["answerable"]))
        if a_has and s_has and a_val != s_val:
            _die("answers/scores %s 的 answerable 冲突:answer 行=%r、score 行=%r"
                 "(两侧必须一致,或只在其中一侧给出)" % (k, a["answerable"], s["answerable"]))
        answerable = a_val if a_has else s_val
        if s is not None:
            judge_error = bool(s.get("judge_error"))
            correct = s.get("correct")
        else:
            judge_error = answerable and not infra
            correct = None
        # a completed ANSWERABLE item with no usable verdict — whether the score row is MISSING or
        # PRESENT-but-undecided (judge_error, or no `correct` at all) — is counted NOT-correct: a
        # trustworthy lower bound. Neither a missing nor an undecided judgment may inflate correctness.
        if answerable and not infra and (judge_error or correct is None):
            judge_error, correct = True, False
        # symmetric lower bound for OOS: a completed out-of-scope item with no abstention verdict
        # (missing score, or a score without `abstained`) counts NOT-abstained — it never inflates
        # abstention_oos (i.e. we don't credit an unverified abstention).
        abstained = (s or {}).get("abstained")
        if (not answerable) and not infra and abstained is None:
            abstained = False
        merged.append({
            "course": a["course"], "model": a["model"], "arm": a["arm"], "item_id": str(a["item_id"]),
            "answerable": answerable, "infra_error": infra, "judge_error": judge_error,
            "correct": correct,
            "faithfulness": (s or {}).get("faithfulness"),
            "hallucinated": (s or {}).get("hallucinated"),
            "abstained": abstained,
            "scored_by": (s or {}).get("scored_by"),
            "cost_usd": cost,
        })
    # scores with no matching answer would be silently ignored — fail loud instead
    orphan = [k for k in score_by if k not in seen]
    if orphan:
        _die("scores 含 %d 条没有对应 answer 的行(每个判分必须对应一个作答),如 %s" % (len(orphan), orphan[0]))
    return merged


def build_summary(merged, primary_course, secondary_course=None, judge_model="unknown"):
    courses = sorted({m["course"] for m in merged})
    models = sorted({m["model"] for m in merged})
    arms = sorted({m["arm"] for m in merged})
    if primary_course not in courses:
        _die("--primary-course %r 不在数据的 course 里(有: %s)" % (primary_course, courses))
    if secondary_course is not None and secondary_course not in courses:
        _die("--secondary-course %r 不在数据的 course 里(有: %s)" % (secondary_course, courses))

    # generic: every course → {model|arm: cell}
    course_matrix = {}
    cells_by = {}
    for m in merged:
        cells_by.setdefault((m["course"], m["model"], m["arm"]), []).append(m)
    for (course, model, arm), items in cells_by.items():
        course_matrix.setdefault(course, {})["%s|%s" % (model, arm)] = _cell(items)

    # cost_per_q: course → arm → mean per-item cost (across models)
    # infra_error 行**不进均值分母**(与正确率同一条诚实规则:它们不是模型答案)——否则一堆 0 成本的
    # 失败行会把「每题成本」摊薄成漂亮数字,报告照登就是虚标。真实总花费仍在 total_cost_usd 里。
    cost_per_q = {}
    cost_acc = {}
    for m in merged:
        ca = cost_acc.setdefault((m["course"], m["arm"]), [0.0, 0])   # 臂键保留(全失败 → None 而非消失)
        if m["infra_error"]:
            continue
        ca[0] += (m.get("cost_usd") or 0)
        ca[1] += 1
    for (course, arm), (tot, n) in cost_acc.items():
        cost_per_q.setdefault(course, {})[arm] = round(tot / n, 4) if n else None

    # 齐平性:主课程各 (model,arm) 格的答题集应一致(「同题跨臂比较」的前提)。不齐平不硬拦
    # (续跑中途单独聚合是合法状态),但要**响亮标出**——headline n_items 是并集,各格分母是自己的子集。
    prim_sets = {}
    for m in merged:
        if m["course"] == primary_course:
            prim_sets.setdefault((m["model"], m["arm"]), set()).add(m["item_id"])
    ragged = len({frozenset(v) for v in prim_sets.values()}) > 1
    if ragged:
        sys.stderr.write("aggregate_matrix: ⚠️ 主课程各格答题集不齐平(%s)——跨臂/跨模型对比要小心,"
                         "n_items 是并集而非公共集\n"
                         % ", ".join("%s|%s=%d" % (k[0], k[1], len(v)) for k, v in sorted(prim_sets.items())))

    n_items = len({m["item_id"] for m in merged if m["course"] == primary_course})
    summary = {
        "n_items": n_items,
        "models": models,
        "arms": arms,
        "courses": courses,
        "judge_model": judge_model,
        "total_cost_usd": round(sum((m.get("cost_usd") or 0) for m in merged), 4),
        "matrix": course_matrix[primary_course],          # report_matrix.py reads this
        "course_matrix": course_matrix,                    # generic, every course
        "cost_per_q": cost_per_q,
        "ragged_matrix": ragged,                           # 主课程各格答题集不齐平(对比要小心)
        "aggregated_by": "aggregate_matrix.py",            # provenance: NOT the published rejudge output
    }
    if secondary_course is not None:
        # renderer-compatible psyc block (psyc|model|arm keys) for an explicitly chosen second course
        summary["psyc"] = {"psyc|%s" % mk: cell for mk, cell in course_matrix[secondary_course].items()}
    return summary


def main(argv=None):
    for s in ("stdout", "stderr"):
        try:
            getattr(sys, s).reconfigure(encoding="utf-8")
        except Exception:
            pass
    ap = argparse.ArgumentParser(
        description="把显式的 answer/score 行聚合成 report_matrix.py 可渲染的 summary.json(纯标准库,无网络/LLM)。")
    ap.add_argument("--answers", required=True, help="作答行 JSONL(题目全集 + 每次作答)")
    ap.add_argument("--scores", required=True, help="判分行 JSONL(每题的判分结果)")
    ap.add_argument("--out", required=True, help="输出 summary.json 路径")
    ap.add_argument("--primary-course", default=None,
                    help="进入 matrix 的主课程(默认取数据里 item 最多的课程)")
    ap.add_argument("--secondary-course", default=None,
                    help="可选:再产出一个 renderer 的 psyc 块(psyc|model|arm)用于该课程")
    ap.add_argument("--judge-model", default="unknown", help="判分模型标注(仅写入 summary,不调用)")
    args = ap.parse_args(argv)

    answers = _load_jsonl(args.answers, REQ_ANSWER, "answers", _alias_answer)
    scores = _load_jsonl(args.scores, REQ_SCORE, "scores", _alias_score)
    if not answers:
        _die("answers 为空——无可聚合的作答")
    merged = merge_rows(answers, scores)

    primary = args.primary_course
    if primary is None:   # default: the course with the most distinct items
        per_course = {}
        for m in merged:
            per_course.setdefault(m["course"], set()).add(m["item_id"])
        primary = max(sorted(per_course), key=lambda c: len(per_course[c]))
    summary = build_summary(merged, primary, args.secondary_course, args.judge_model)

    out_dir = os.path.dirname(os.path.abspath(args.out))
    if out_dir:
        os.makedirs(out_dir, exist_ok=True)
    with open(args.out, "w", encoding="utf-8") as f:
        json.dump(summary, f, ensure_ascii=False, indent=2)
    print("[+] summary: %s(%d 课程 / %d 模型 / %d 臂;主课程 %s,n_items=%d)"
          % (args.out, len(summary["courses"]), len(summary["models"]), len(summary["arms"]),
             primary, summary["n_items"]))
    return 0


if __name__ == "__main__":
    sys.exit(main())
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall