Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/testing-audit.en.md

View on GitHub
# Testing & Benchmark Audit

English · [中文](testing-audit.md)

> This is an **honest snapshot of the current state**. Its purpose is to spell out "what we actually test and what we don't" before we invest in expensive tests.
> It is not a marketing document. Its conclusions point to the PR roadmap (see the end). Current progress: **T1** (audit doc + consistency guard), **T2** (Tier 2 deterministic behavior-smoke layer), **T3/T3.1** (committed aggregator `aggregate_matrix.py` + fixture pipeline + judging↔aggregation bridge) and **T4** (Tier 4 long-horizon drift **deterministic replay harness** [`drift/`](../drift/)) have landed; T2's **real-LLM behavior-smoke wiring is hooked up** (B2: behavior_smoke `--llm` single-turn + stub-based deterministic tests), and T4's **real-LLM long-session** runner is also ready; real paid runs and **T5** remain opt-in (the full published matrix still needs private/paid artifacts). This file is a living snapshot maintained across PRs; it does not run paid benchmarks.

---

## 1. Test tiers and what actually runs

| Tier | What it is | In CI? | Cost | Reality |
| :-- | :-- | :--: | :-- | :-- |
| **Tier 0** | `python -m unittest discover -s tests` (stdlib unit/static test suite) | ✅ Ubuntu + Windows × Py 3.8 + 3.12 | $0, pure standard library, no network, no API key | The **only** tier CI actually runs |
| **Tier 1** | `python scripts/validate_workspace.py <workspace>` (a built workspace's structure/schema/provenance labels/path safety) | 🟡 integration test runs in CI with Tier 0 | $0 | The validator logic is covered by Tier 0 unit tests; **since B6** there is also a deterministic `ingest → validate` integration test (`tests/test_ingest_validate_integration.py`: it actually runs both CLIs; real ingest output must pass validation, and tampered/missing-image/bad-JSON workspaces must exit with 1/2). It reaches CI through the root tests, with no separate Actions step (CI configuration deliberately not added) |
| **Tier 2** | **Behavior smoke**: deterministic mock layer (see [`behavior_smoke/`](../behavior_smoke/)) | ✅ mock in CI | Near $0 | The deterministic layer has landed; **since T5c there is an opt-in real-agent smoke runner** ([`drift/run_live_smoke.py`](../drift/run_live_smoke.py): drive → record → T4 scoring in one command; env-gated, not in CI); **behavior_smoke `--llm` single-turn is wired up (B2), and the wiring is tested deterministically with a stub; real paid runs remain opt-in and have not yet been run**, see §4 |
| **Tier 3** | Full benchmark (`gen.py` → judging → matrix report) | ❌ | **Expensive**: a single-round matrix costs roughly tens of dollars / several hours | Manual, ad hoc |
| **Tier 4** | Long-horizon drift | 🟡 replay layer is in (root-level) tests | Replay $0; real LLM costs days of quota | **The deterministic replay harness has landed** ([`drift/`](../drift/): replays scripted transcripts; pure stdlib; zero cost); **real-LLM long sessions remain opt-in, unimplemented, and not in CI** |

**CI status**: `.github/workflows/ci.yml` has a single step, `python -m unittest discover -s tests -v`, so it runs only Tier 0 (**which now includes `tests/test_behavior_smoke.py`, Tier 2's deterministic mock layer**); all of it is zero cost. The README has been corrected accordingly to say "CI actually runs only Tier 0". The Tier 1 validator's logic is covered by Tier 0 unit tests on `tests/fixtures/`; **since B6**, the deterministic `ingest → validate` integration test (which actually runs both CLIs) also lives in the root `tests/` and runs in CI with Tier 0 (still no separate Actions step; CI configuration deliberately not added).

---

## 2. What Tier 0 checks

`tests/` holds a stdlib unit/static test suite (run `python -m unittest discover -s tests -v` to see the current count and details). It covers:

- **Workspace validator**: structure, question-bank schema (6 question types), provenance labels (`source` enum + mandatory `ai_generated` flag), **path safety** (symlinks, directory traversal, backslashes, Markdown links, NaN), exit codes 0/1/2.
- **ingest end to end**: generated files, anchor replacement, loud errors on bad input, refusal of traversal/duplicate-name wikis, acceptance of 6 question types, warnings for missing answers, rerun/`--force` backups.
- **Language policy / bilingual control layer**: canonical provenance-label wording, English control-layer headings, Simplified Chinese as the student-facing default, no vague wording.
- **Skill collection self-consistency**: `skills/` frontmatter (name/description/license), AGENTS fallback invariants, docs present, confusion-tracker merged into `skills/`.
- **Runtime wording**: no version numbers / no abstract "protocol" wording.

**What they are**: all of these are **structure / schema / file-content** assertions + script tests of `ingest`/`validate_workspace`. They prove that "the instructions and artifacts exist and are well formed"; they **do not execute real agent behavior**.

## 3. What Tier 1 checks

`scripts/validate_workspace.py` runs zero-cost validation on an **already built exam-prep workspace**: directory structure, `quiz_bank.json` schema (question type/options/answer within the options/subjective-question keywords/diagram_type/true_false boolean/source enum/`ai_generated` flag), `references/wiki/` path safety, and progress-file consistency; exit code `0` pass / `1` errors / `2` fatal (broken structure or invalid JSON). It is driven by Tier 0 unit tests on fixtures; **since B6** there is also `tests/test_ingest_validate_integration.py`, which actually runs this CLI on **real ingest output** (happy 0 / tampered 1 / bad JSON 2) and runs in CI with the root tests.

## 4. Tier 2 behavior smoke: the deterministic layer has landed; real-LLM behavior is still opt-in (not in CI)

This PR (T2) landed Tier 2's **deterministic mock layer**: a small self-authored fixture (`benchmark/behavior_smoke/fixtures/mini_course`, which passes Tier-1 validation and covers all 6 question types) + a set of **deterministic detectors** that assert the following behaviors on mock artifacts, all in CI and at zero cost:

- quiz_bank-only questions (question IDs must ∈ the bank; invented IDs are caught)
- canonical 🟢/🟡/⚠️ provenance-label output
- hint / skip / mistake logging (escape hatches + mistake rows written)
- conceptual confusion tracking (confusion rows written to progress)
- study_progress checkpoint resume (current phase read back)
- write-to-disk fallback without a Python environment (hand-written workspace passes validation)
- beginner key-question walkthrough (upgraded to the A5 seven-step template)

The phases of the A line added **5 more Tier 2 deterministic behavior scenarios** (`behavior_smoke/scenarios.json`, registered in [`coverage-matrix.en.md`](coverage-matrix.en.md) when B1 wrapped up):

- Visual questions: prompt-image gate (`visual_first_assets`, P0-V1/A1)
- Scope filtering + out-of-scope override announcement (`scope_override`, A2)
- Seven-step walkthrough template + per-item source block (`teaching_template`, A5)
- ≤1 day skips clarification/preference/reflective follow-up questions but allows bank-verified checkpoints; only an explicit `no_questions` forbids all interactive questions and caps completion at `covered_unverified` (`time_budget_no_questions`, A6)
- Knowledge points outside the window must really be rechecked (`knowledge_window_recheck`, A6; >7 days requires an actual quiz question)

There is also **1 Tier 4 long-session replay scenario** (`benchmark/drift/scenarios/mode_urgent_no_questions.json`, **not** a Tier 2 smoke): the student in this scenario has explicitly asked not to be asked questions, so it verifies zero interactive questions (`urgent_mode_questions`) + that the urgent opening is inferred and persisted (`urgent_mode_persisted`). It does not mean that an ordinary ≤1 day budget forbids bank checkpoints.

**Still honest**: the deterministic layer only proves that the detectors hold on the **expected artifacts**; it **does not prove that a real LLM agent will produce these behaviors**. **The behavior_smoke `--llm` single-turn real-agent smoke is wired up** (B2: it drives each scenario, applies the same detectors as `--mock`, and honestly reports state-type scenarios as SKIP; `tests/test_behavior_smoke_live.py` verifies the wiring deterministically with a stub). Real paid runs remain opt-in and not in CI (`RUN_SKILL_BEHAVIOR_LLM=1` or `--agent-cmd`) and have not yet been run against a real model; **LLM Wiki lazy loading** and **running the algorithm before drawing** remain best-effort / uncovered. See [`../behavior_smoke/README.en.md`](../behavior_smoke/README.en.md) and [`coverage-matrix.en.md`](coverage-matrix.en.md) for details.

## 5. Tier 3 full benchmark: manual and expensive

A real-LLM matrix across `course × model × arm`; a single round costs roughly tens of dollars / several hours. It measures **grounding / hallucination / out-of-scope abstention in grounded Q&A**, not the interactive tutoring flow. Trigger it by hand only when you "deliberately changed skill behavior / added new course materials / are about to release"; it should not go into CI or run for every small change.

## 6. Tier 4 long-horizon drift: deterministic replay has landed; real-LLM long sessions are still future work

Goal retention / plan adherence / invention rate / checkpoint recovery / provenance labels / progress persistence in long conversations (dozens of turns + mid-session distractions). **T4 has landed a deterministic replay harness** [`drift/`](../drift/): it replays scripted multi-turn transcripts + workspace snapshots (self-authored, non-copyrighted fixtures) and measures the dimensions above **deterministically** against thresholds: pure stdlib, zero cost, in root-level tests (`python benchmark/drift/run_drift.py --all`). **But it replays scripted sessions and does not run a real agent**: it measures "whether a recorded session drifted"; it does **not** prove that an online model won't drift. **Real-LLM long sessions remain opt-in (`--llm` + `RUN_SKILL_DRIFT_LLM=1`), unimplemented, never return success, and are not in CI**; the full long-horizon LLM benchmark (costing days of quota) is still future work.

---

## 7. Benchmark pipeline status (stated honestly)

- **Two runners coexist, with different arm definitions**:
  - `run_benchmark.py` is the **older two-arm scaffold** (`baseline` / `skill`); it cannot resume and **discards cost**.
  - `gen.py` is the **newer matrix answer-generation** path; it supports all four arms (`closedbook` / `material` / `rawfiles` / `skill`), **can resume, and records cost per cell**. But it is an **incremental gap-filler ordered by feasibility, not a from-scratch full generator**: `build_tasks()` actually queues only rawfiles (both courses), PSYC's closedbook/skill, **reruns of only the failed algo `material` cells** (read from the existing `results/matrix/answers.jsonl`), and PSYC material. It **depends on an existing `answers.jsonl`** (`build_tasks()` opens it unconditionally), and that file is **not committed**, so on a clean checkout `python benchmark/gen.py` fails immediately with `FileNotFoundError` and **cannot run at all**. It also **hard-codes the MIT/PSYC item prompts and workspaces**, accepts only `--limit`, and does not read the user's `config.json`/`items.jsonl`. Therefore **the published matrix can neither be run from a clean checkout nor be reproduced from scratch with this repository's scripts alone**. This is a reproducibility gap separate from "the missing aggregator".
- **The three primary control arms are standardized as `closedbook` / `rawfiles` / `skill`**:
  - `closedbook`: no course materials at all.
  - `rawfiles`: raw files + a general-purpose agent, no skill installed. **The fairest control.**
  - `skill`: the built wiki + quiz_bank, using this skill.
  - **Legacy/stress arm `material` / dump-all**: the whole course's full text stuffed into a single prompt. **Not a primary control**; kept as a **stress footnote** (in practice it overwhelms weak models and triggers context/usage limits that crash the run).
- **`report_matrix.py` only renders; it does not compute**: by default it renders `results/matrix/summary.json`. Since T3 it supports `--summary <file> --out-dir <dir>` and can render an **explicit** summary (it is no longer forced to render only the committed one).
- **T3 filled in the `summary.json` aggregator**: `benchmark/aggregate_matrix.py` (**new in T3**, pure standard library) aggregates a `summary.json`-compatible matrix summary from **explicit** answer/score rows, and there is a fixture-level reproducible pipeline (see [`matrix_pipeline.en.md`](matrix_pipeline.en.md)). **But the published MIT/PSYC `summary.json` is still a precomputed artifact**: the full matrix still depends on **private/intermediate artifacts + paid model runs** (real answer logs and private gold labels are not committed), and the aggregator by itself does **not** reproduce the published numbers. (Also note: after re-judging, `rejudge.py` writes `summary_corrected.json`, whose structure is **nested** `algo` / `psyc` blocks, **not** the top-level `matrix`/`n_items` shape that `report_matrix.py` reads, so it **cannot** be rendered directly with `--summary summary_corrected.json`. It must first be converted to the top-level matrix shape, and re-judged results do not enter the matrix report automatically. **T3.1 built this bridge**: `rejudge.py --scores-out` (**`--answers-out` must be given too**) exports each item's score, at zero cost and without calling an LLM, as score/answer rows that `aggregate_matrix.py` can read (answer rows carry `status`/`cost_usd`), so `judge/rejudge → aggregate_matrix → report_matrix` becomes a committed, explicit path you can follow. The default behavior is unchanged, and the export writes only to explicit paths.)
- **Judging already has a deterministic fast path**: out-of-scope abstention / numeric items / exact lexical matches are decided before the LLM judge is called; only undecided factual/definition items go to the LLM judge.
- **`--mock` only validates the pipeline**: the mock answers and the mock judge are both preset. They can verify that the pipeline connects, but they **cannot catch regressions in real judging quality**.

---

## 8. Coverage gaps for new skill capabilities

In one sentence: **structure / schema / instruction text are thoroughly covered statically; behavior is barely verified by execution at all.** For a per-capability comparison see [`coverage-matrix.en.md`](coverage-matrix.en.md). Capabilities that need to be tested at the **behavior** level (and currently are not): lazy loading, quiz_bank-only questions, actual grading of the six question types, running the algorithm before drawing, beginner walkthroughs, hint/skip/mistake logging, confusion tracking, checkpoint resume, runtime provenance labels, and the no-Python fallback.

---

## 9. Token-saving principles

1. **Deterministic first**: whether provenance labels appear, whether every drawn question is in the bank, whether progress was written, whether only one chapter was read, whether diagram questions carry a render_hint: these are decided with **structural assertions on the artifact files** and **need no LLM judge**.
2. **Cache generation and judging**: `gen.py` already caches answers + cost per cell and can resume; judging results should also be cached (ideally keyed by a hash of the judge prompt, so that a change to the prompt template correctly invalidates the cache).
3. **Use the LLM judge only for undecided items**: call the judge only for factual/definition items that the deterministic fast path cannot settle.
4. **Keep Tier 2 small and mostly deterministic**: about a dozen behavior scenarios, a single cheap model (Haiku), mostly structural assertions, and only a very few that need one cheap grounding call.
5. **Paid tiers must be triggered by hand**: the full benchmark (Tier 3) and **real-LLM** long-horizon drift (the `--llm` part of Tier 4) never go into CI and never run for small changes; Tier 4's **deterministic replay layer** is zero cost and already in the root-level tests.

---

## PR roadmap (T-track)

- **T1** ✅: audit doc + coverage matrix + benchmark doc consistency guard (zero cost, docs/tests only).
- **T2** ✅ (deterministic layer): Tier 2 behavior smoke: self-authored non-copyrighted fixture + harness + deterministic detectors (in CI); the `--llm` single-turn real-agent smoke **wiring is hooked up (B2, stub-based deterministic tests)**; **real paid runs remain opt-in, are not in CI, and have not yet been run against a real model**.
- **T3** ✅ (aggregation layer): committed `summary.json` aggregator [`aggregate_matrix.py`](matrix_pipeline.en.md) + fixture-level reproducible pipeline + `report_matrix.py --summary`. **The full published matrix still depends on private/paid artifacts**; benchmark caching/resume/cost logging are later increments.
- **T4** ✅ (deterministic replay layer): Tier 4 long-horizon drift [`drift/`](../drift/): self-authored non-copyrighted fixtures that replay scripted transcripts + snapshots and deterministically measure goal retention/plan adherence/invention rate/checkpoint recovery/provenance labels/progress persistence/out-of-chapter wiki reads (in root-level tests); **real-LLM long sessions are opt-in, not in CI, and not yet implemented**.
- **T5**: judge calibration (larger kappa sample, add near-miss out-of-scope probes, cross-family judges, fix numeric extraction).
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall