Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/gen.py

View on GitHub
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Resumable, quota-aware answer GENERATOR for the benchmark (real `claude -p`).

Fills the gaps the first run left:
  * algo (6.006) `material` arm — re-generate ONLY the items that errored out (rate-limit / API
    error) the first time; the valid material answers are kept.
  * psyc (PSYC 110) — generate all 3 arms x 3 models for real (the old results/raw.jsonl was MOCK).

Reality: the `material` arm dumps the whole course (~100K tokens for 6.006, ~230K for PSYC), which
burns the Claude subscription's 5-hour cap fast and may exceed context. So this is RESUMABLE:
  - successful answers are cached (gen_cache.jsonl) and appended to gen_answers.jsonl;
  - transient errors ("temporarily limiting" / API error) are retried with backoff;
  - a HARD subscription limit ("hit your limit / resets ...") is NOT retried — after a few in a row
    the pass STOPS cleanly so it can be resumed after the quota resets (just run gen.py again).

Tasks are ordered by feasibility: PSYC closed-book+skill (small) -> algo material reruns -> PSYC
material (largest, may be infeasible). Run repeatedly until it reports 0 remaining.

    python gen.py            # one resumable pass
    python gen.py --limit 3  # smoke test: only the first 3 tasks
"""
import os
import re
import sys
import json
import time
import argparse
import subprocess

for _s in ("stdout", "stderr"):
    try:
        getattr(sys, _s).reconfigure(encoding="utf-8")
    except Exception:
        pass

HERE = os.path.dirname(os.path.abspath(__file__))
GEN_ANS = os.path.join(HERE, "results", "matrix", "gen_answers.jsonl")
GEN_CACHE = os.path.join(HERE, "results", "matrix", "gen_cache.jsonl")
GEN_PROG = os.path.join(HERE, "results", "matrix", "gen_progress.txt")
MODELS = ["opus", "sonnet", "haiku"]

COURSES = {
    "algo": {"combined": "materials/algorithms_mit6006/_combined.txt",
             "skill_ws": "skill_workspace/mit6006_full", "raw_ws": "skill_workspace/rawfiles_algo",
             "qfile": "items/items_algo_full_q.jsonl"},
    "psyc": {"combined": "materials/psych_yale_psyc110/_combined.txt",
             "skill_ws": "skill_workspace/psyc110_full", "raw_ws": "skill_workspace/rawfiles_psyc",
             "qfile": "items/items_psyc_full_q.jsonl"},
}

CLOSEDBOOK = ("请仅凭你自己的知识回答下面的问题。如果你不确定或不知道,请直接回答「材料中未涵盖」或「不确定」,"
              "不要编造。\n\n问题:{q}\n\n请直接给出简洁答案。")
MATERIAL = ("请只依据下面的【课程材料】回答问题;材料中没有的内容,请直接回答「材料中未涵盖」,不要编造。\n\n"
            "【课程材料】\n{material}\n\n【问题】{q}\n\n请直接给出简洁答案。")
SKILL = ("你是备考教练。请依据本工作区已建立的 references/wiki/ 知识库回答问题;材料未涵盖的内容请回答"
         "「材料中未涵盖」,不要现场编造或重新推导。\n\n【问题】{q}\n\n请直接给出简洁答案。")
# 公平对照:无 skill,但 agent 能按需读取本文件夹里的原始讲义/习题(用 Read/Glob/Grep),而不是一次性全塞进提问
RAWFILES = ("本文件夹里是这门课的原始材料文件(讲义、习题)。请先用工具查阅相关文件,再据此回答问题;"
            "材料未涵盖的内容请回答「材料中未涵盖」,不要编造或凭记忆作答。\n\n【问题】{q}\n\n请直接给出简洁答案。")

_HARD = ("hit your limit", "usage limit", "resets ")
_TRANSIENT = ("temporarily limiting", "rate limited", "api error", "overloaded")


def load_q(path):
    return {d["id"]: d["question"] for d in (json.loads(l) for l in open(path, encoding="utf-8") if l.strip())}


def read(path):
    p = os.path.join(HERE, path)
    return open(p, encoding="utf-8").read() if os.path.exists(p) else ""


def classify(text):
    t = (text or "").lower()
    if any(h in t for h in _HARD):
        return "hard"
    if any(x in t for x in _TRANSIENT):
        return "transient"
    return "ok"


def parse_stream_events(stdout_text):
    """Parse `--output-format stream-json --verbose` output → (result, cost, files_opened).
    Pure function (offline-testable). files_opened = every file_path/path/pattern string the
    agent passed to Read/Glob/Grep tool calls, in call order, deduped — the retrieval trace
    (v4-P3 缺口8:不记轨迹就永远测不了「到底翻了哪章」)."""
    result, cost, files, seen = "", None, [], set()
    for line in (stdout_text or "").splitlines():
        line = line.strip()
        if not line:
            continue
        try:
            ev = json.loads(line)
        except json.JSONDecodeError:
            continue
        if ev.get("type") == "result":
            result = ev.get("result", "") or result
            cost = ev.get("total_cost_usd", cost)
        elif ev.get("type") == "assistant":
            for block in (ev.get("message") or {}).get("content") or []:
                if block.get("type") == "tool_use" and block.get("name") in ("Read", "Glob", "Grep"):
                    inp = block.get("input") or {}
                    # Grep 的 pattern 是内容搜索正则、不是路径——记进 files_opened 会让
                    # retrieval_eval 把 "lecture02" 这类搜索词当成「翻开了第 2 章」,虚标
                    # recall(Codex r2)。Glob 的 pattern 才是路径面;Grep 只记 path 过滤器。
                    keys = {"Read": ("file_path",), "Glob": ("pattern", "path"),
                            "Grep": ("path",)}[block["name"]]
                    for k in keys:
                        v = inp.get(k)
                        if isinstance(v, str) and v and v not in seen:
                            seen.add(v)
                            files.append(v)
    return result, cost, files


def _run_claude_impl(prompt, model, cwd=None, skill=False, timeout=900, trace=False):
    # Pass the prompt via STDIN, not argv — the material arm dumps a ~100-230K-char course, which
    # blows Windows' ~32K command-line limit (WinError 206) if passed as an argument.
    if trace:
        args = ["claude", "-p", "--output-format", "stream-json", "--verbose", "--model", model]
    else:
        args = ["claude", "-p", "--output-format", "json", "--model", model]
    if skill:
        args += ["--allowedTools", "Read", "Glob", "Grep"]
    try:
        p = subprocess.run(args, cwd=os.path.join(HERE, cwd) if cwd else HERE, input=prompt,
                           capture_output=True, text=True, encoding="utf-8", timeout=timeout)
        if trace:
            result, cost, files = parse_stream_events(p.stdout)
            if result:
                return result, cost, files
            return (p.stdout or p.stderr or "").strip()[:2000], None, files
        try:
            data = json.loads(p.stdout)
            return data.get("result", "") or "", data.get("total_cost_usd"), None
        except json.JSONDecodeError:
            return (p.stdout or p.stderr or "").strip(), None, None
    except subprocess.TimeoutExpired:
        return "TIMEOUT", None, None
    except Exception as e:                             # never let one bad call kill the whole pass
        return f"API Error: {e}", None, None


def run_claude(prompt, model, cwd=None, skill=False, timeout=900):
    """(out, cost) — the stable 2-tuple every existing caller (run_matrix judge path 等) unpacks.
    轨迹要走 run_claude_traced;本函数**永不**因环境变量改变返回形状(Codex r1)。"""
    out, cost, _files = _run_claude_impl(prompt, model, cwd, skill, timeout, trace=False)
    return out, cost


def run_claude_traced(prompt, model, cwd=None, skill=False, timeout=900):
    """(out, cost, files_opened) — stream-json 工具轨迹版(检索评测 recall@k 的数据源)。"""
    return _run_claude_impl(prompt, model, cwd, skill, timeout, trace=True)


def generate_one(course, model, arm, qid, q, combined):
    """(out, cost, files_opened|None)。EXAMPREP_TRACE=1 只在**作答生成**路径开启轨迹——
    判分等其它 run_claude 调用点不受影响(返回形状恒定)。"""
    trace = os.environ.get("EXAMPREP_TRACE") == "1"
    call = run_claude_traced if trace else (lambda *a, **k: run_claude(*a, **k) + (None,))
    if arm == "closedbook":
        return call(CLOSEDBOOK.format(q=q), model)
    if arm == "material":
        return call(MATERIAL.format(material=combined, q=q), model)
    if arm == "rawfiles":
        return call(RAWFILES.format(q=q), model, cwd=COURSES[course]["raw_ws"], skill=True)
    return call(SKILL.format(q=q), model, cwd=COURSES[course]["skill_ws"], skill=True)


def build_tasks():
    """Ordered by feasibility. Returns list of (course, model, arm, id)."""
    tasks = []
    qalgo = load_q(os.path.join(HERE, COURSES["algo"]["qfile"]))
    qpsyc = load_q(os.path.join(HERE, COURSES["psyc"]["qfile"]))
    # 0) rawfiles arm — fair no-skill agentic baseline (read raw files on demand), BOTH courses x 3 models
    for course, q in (("algo", qalgo), ("psyc", qpsyc)):
        for m in MODELS:
            for qid in q:
                tasks.append((course, m, "rawfiles", qid))
    # 1) PSYC closed-book + skill (small prompts) — the feasible, high-value real second course
    for arm in ("closedbook", "skill"):
        for m in MODELS:
            for qid in qpsyc:
                tasks.append(("psyc", m, arm, qid))
    # 2) algo material reruns — only the items that errored the first time
    ERR = ("hit your limit", "api error", "rate limited", "temporarily limiting", "usage limit")
    for r in (json.loads(l) for l in open(os.path.join(HERE, "results/matrix/answers.jsonl"),
                                          encoding="utf-8") if l.strip()):
        if r["tag"] == "matrix" and r["arm"] == "material":
            a = (r.get("answer") or "").lower()
            if any(e in a for e in ERR):
                tasks.append(("algo", r["model"], "material", r["id"]))
    # 3) PSYC material (largest; may exceed context) — attempted last
    for m in MODELS:
        for qid in qpsyc:
            tasks.append(("psyc", m, "material", qid))
    return tasks


def load_cache():
    cache = set()
    if os.path.exists(GEN_CACHE):
        for d in (json.loads(l) for l in open(GEN_CACHE, encoding="utf-8") if l.strip()):
            cache.add(tuple(d["key"]))
    return cache


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--limit", type=int, default=0)
    args = ap.parse_args()

    qcache = {c: load_q(os.path.join(HERE, COURSES[c]["qfile"])) for c in COURSES}
    combined = {c: read(COURSES[c]["combined"]) for c in COURSES}
    tasks = build_tasks()
    if args.limit:
        tasks = tasks[:args.limit]
    done = load_cache()
    cf = open(GEN_CACHE, "a", encoding="utf-8")
    af = open(GEN_ANS, "a", encoding="utf-8")

    n_ok = n_fail = hard_streak = 0
    t0 = time.time()
    todo = [t for t in tasks if t not in done]
    print(f"[gen] 任务 {len(tasks)},已完成 {len(tasks)-len(todo)},本次待跑 {len(todo)}")
    for i, (course, model, arm, qid) in enumerate(todo, 1):
        q = qcache[course].get(qid, "")
        ans, cost, files = "", None, None
        for attempt in range(3):                      # retry transient errors with backoff
            ans, cost, files = generate_one(course, model, arm, qid, q, combined[course])
            kind = classify(ans)
            if kind == "ok":
                break
            if kind == "hard":
                break                                 # don't retry a hard subscription limit
            time.sleep(5 * (attempt + 1) ** 2)        # 5s, 20s, 45s
        kind = classify(ans)
        if kind == "ok" and ans.strip():
            key = [course, model, arm, qid]
            cf.write(json.dumps({"key": key}, ensure_ascii=False) + "\n"); cf.flush()
            row = {"course": course, "model": model, "arm": arm, "id": qid,
                   "answer": ans, "cost": cost}
            if files:                                 # 轨迹开着才有——检索评测的数据源
                row["files_opened"] = files
            af.write(json.dumps(row, ensure_ascii=False) + "\n"); af.flush()
            n_ok += 1; hard_streak = 0
        else:
            n_fail += 1
            if kind == "hard":
                hard_streak += 1
        msg = f"{i}/{len(todo)} ok={n_ok} fail={n_fail} hard_streak={hard_streak} {int(time.time()-t0)}s | last={course}/{model}/{arm}/{qid}:{kind}"
        with open(GEN_PROG, "w", encoding="utf-8") as pf:
            pf.write(msg + "\n")
        if i % 5 == 0:
            print("  " + msg)
        if hard_streak >= 6:                          # quota exhausted -> stop cleanly, resume later
            print(f"\n[gen] 连续 {hard_streak} 次撞订阅配额上限,本次到此停(已存进度)。配额恢复后再跑 gen.py 续。")
            break
    cf.close(); af.close()
    remaining = len([t for t in tasks if t not in load_cache()])
    print(f"\n[gen] 本次完成 {n_ok},失败 {n_fail},剩余 {remaining}。用时 {int(time.time()-t0)}s")
    print(f"      -> {GEN_ANS}")


if __name__ == "__main__":
    main()
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall