Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/docs/coverage-matrix.en.md

View on GitHub
# Feature × Tier Coverage Matrix

English · [中文](coverage-matrix.md)

> Companion to [`testing-audit.md`](testing-audit.en.md). In one sentence: **many capabilities are tested statically at the structure/instruction layer but have not been executed and verified at the behavior layer.**

**Legend**: ✅ covered 🟡 partial (structure/schema layer, **or** Tier 2 deterministic mock detection—not real LLM behavior) ❌ not covered — not applicable

> **Tier 2 column note**: 🟡 means this PR’s (T2) **deterministic mock detector** covers artifact assertions for the behavior (in CI, zero cost). It **does not mean** real LLM behavior has been verified. Real-Agent smoke is opt-in and excluded from CI; see [`../behavior_smoke/`](../behavior_smoke/).
> **B2 is wired**: `behavior_smoke --llm` single-turn real-Agent smoke is no longer a skeleton. It drives `claude -p` per scenario (or an `--agent-cmd` stub), applies the **same detector function** as `--mock` to the response (the primary positive check for each scenario; mock-only extra negative/variant assertions validate the detector and are not rerun live), writes a transcript, and emits metrics. State/file scenarios that a one-shot `-p` cannot verify are honestly reported as SKIP. This is opt-in, off by default, and excluded from CI; `tests/test_behavior_smoke_live.py` verifies the wiring deterministically with a stub Agent. Every table note saying “real LLM … unverified (opt-in)” refers to this route.
>
> **Tier 4 column note**: 🟡 means PR T4’s **deterministic replay harness** in [`../drift/`](../drift/) covers the behavior across **multi-turn, long-running conversations** by replaying scripted transcripts plus snapshots in root tests at zero cost. It likewise **does not mean** real LLM long-session behavior has been verified. Real-Agent long sessions are opt-in, unimplemented, and excluded from CI.

| Feature | Tier 0 unit/static | Tier 1 workspace validator | Tier 2 behavior smoke | Tier 3 full benchmark | Tier 4 long-horizon drift | Current gap | Cheap metric that could be added |
| :-- | :--: | :--: | :--: | :--: | :--: | :-- | :-- |
| Goal retention | ❌ | — | ❌ | ❌ | 🟡 (replay: rate of topic drift/refusal of original plan) | Real-LLM long sessions unverified (opt-in) | Count topic-drift phrases in scripted long sessions |
| Plan adherence | ❌ | 🟡 (progress `current_phase ∈ plan`) | ❌ | ❌ | 🟡 (replay: unauthorized phase deletion/replacement/addition count) | Real-LLM long sessions unverified (opt-in) | `study_plan` snapshot diff |
| LLM Wiki lazy loading | ❌ (instruction text only) | — | 🟡 (best-effort placeholder `count_wiki_reads`, not asserted in CI) | 🟡 (skill arm uses the wiki but does not test laziness) | 🟡 (replay: cross-chapter wiki reads / unique file count) | A real LLM is required to verify “read only the relevant chapter” | File-read call count in a scripted transcript = one chapter |
| One-command ingestion cold start | ✅ (ingestion end-to-end unit test) | 🟡 (validates artifact structure) | — | ❌ | — | Already covered reasonably well | Keep as is |
| Write-to-disk fallback without Python | ❌ | — | 🟡 (mock `no_python_fallback`: hand-written workspace passes Tier-1 validation) | ❌ | — | A true end-to-end run with Python disabled remains untested | Deterministic: hand-written workspace → `validate_workspace` exits 0 |
| `quiz_bank`-only questions | 🟡 (question-bank schema) | 🟡 (schema) | 🟡 (mock `quiz_bank_only`: question ID ∈ bank + invented IDs detected) | ❌ | 🟡 (replay: invention rate across session / cross-phase questions) | Real-LLM question selection unverified (opt-in) | Scripted question selection → assert every ID ∈ bank |
| Six question types | ✅ (schema accepts all six) | ✅ (schema enforces them) | 🟡 (fixture covers and validates all six) | ❌ (item set contains only factual/definition/numeric) | — | Real-LLM per-type asking/grading unverified | One grading smoke item of each type from a fixed bank |
| Run algorithm before drawing a diagram question | ❌ | 🟡 (`diagram_type` warning) | ❌ (fixture has `render_hint`, but no “compute before drawing” behavior scenario) | ❌ | — | Behavior untested | Transcript computes first, then renders |
| Beginner key-question walkthrough (→ A5 seven-step template) | ❌ | — | 🟡 (mock `zero_basic_key_question`: good case uses all ①–⑦ + source block; legacy two-part form without ①–⑦ fails) | ❌ | — | Real-LLM walkthrough unverified (opt-in) | Scripted “key question” request → assert seven steps + source block |
| Seven-step walkthrough template + per-item source block (A5) | 🟡 (`teaching_template` detector tests: circled numbers / heading boundaries / per-item segmentation / ⑦ provenance) | — | 🟡 (mock `teaching_template`: ①–⑦ complete and ordered, ② before ④, ⑦ reaches wiki/page, per-item source block `题目来源|答案来源|🟢/🟡/⚠️`, double ⚠️ for AI answers, default stop at source block, opt-in closing block; more than ten negative cases caught) | ❌ | ❌ | Real-LLM teaching output unverified (opt-in) | Scripted teaching → assert seven steps + source block + ⚠️ |
| Visual-question prompt-image gate (P0-V1 / A1) | ✅ (`show_question_assets`/index-script unit tests) | ✅ (validator rejects a workspace where a visually required item lacks a prompt-side asset) | 🟡 (mock `visual_first_assets`: prompt image first + 11 negative cases—answer image first/answer leak/unlabeled/late/unsafe path/path printed only—all caught) | ❌ | ❌ | Real-LLM rendering order unverified (opt-in) | Scripted visual question → assert prompt image precedes explanation/answer image |
| Scope filtering + out-of-scope override announcement (A2) | 🟡 (`select_questions` selection by `source_type` unit test) | 🟡 (schema: `source_type` label) | 🟡 (mock `scope_override`: verbatim `⚠️ 临时覆盖…范围偏好` must precede the first out-of-scope question; negative case caught) | ❌ | ❌ | Real-LLM scope switching unverified (opt-in) | Scripted out-of-scope question → assert override announcement precedes it |
| Learning mode × time budget (A6) | ✅ (`update_progress` set/normalization/legacy-four-mode migration/unknown-value preservation tests; urgent-alias normalization; explicit `no_questions` cap) | 🟡 (state schema: mode/time_budget/preferences) | 🟡 (mock `time_budget_no_questions`: allows a real-bank checkpoint under ordinary ≤1 day, catches clarification/preference/reflection questions; zero interactive questions under explicit `no_questions`) | ❌ | 🟡 (replay `mode_urgent_no_questions`: explicit no-questions scenario requires `urgent_mode_questions`=0 + `urgent_mode_persisted`=1) | Real-LLM cadence by time tier unverified (opt-in) | Two scripted conversations: ordinary ≤1 day allows bank checkpoint; explicit no-questions has zero interactive questions + persistence |
| Combined first language question + dispatch (A8b) | 🟡 (`tests/test_study_state.py` language normalization + `tests/test_language_purity.py` bilingual composition + EN vocabulary/purity + `tests/test_language_policy.py` dispatch pins) | ❌ | 🟡 (mock `language_first_ask`: first question contains three language rows; accepts three `set` flags in any order, catches noncanonical values; urgent defaults + never infers bilingual) | ❌ | ❌ | Long-session consistency of EN output untested | Add an EN variant to a drift scenario (future item, independent of the delivered A8c EN entry surfaces) |
| Four-part cheatsheet layout (phase 6 C1: 必背→例题→例题解答→要点解释, examples visual-first) | 🟡 (`tests/test_language_policy.py` section pins + control-layer section pins + localization pins) | ❌ | ❌ | ❌ | ❌ | Static contract only; no output behavior detector | Low priority: register if a cheatsheet smoke scenario is added later |
| Knowledge window (A6, 3–7 days/>7 days) | ✅ (`window-add`/`window-set-status` tests: chapter-scoped lookup, multi-chapter ambiguity fail-loud, idempotent backfill, lossless init round trip) | 🟡 (state schema: knowledge_window object array) | 🟡 (mock `knowledge_window_recheck`: out-of-window knowledge must be genuinely rechecked; >7 days must test with a question; negation/default closing does not count—**wording only, no state row written**) | ❌ | 🟡 (B3 replay `window_persist`: long sessions register `window_rows_added`≥2 while teaching and change status without deleting rows, `window_rows_lost`=0; silent row loss fails; malformed row without `point` fails loud) | Real-LLM knowledge-window transitions unverified (opt-in) | Scripted out-of-window point → assert recall question or test |
| Difficulty scoring + hard-question selection (A7) | ✅ (`score_difficulty` tests for every signal dimension [cross-knowledge/structure/needs figure/multiple pages/chapter position/question type] + atomic rewrite/idempotence/dry-run/symlink rejection; `select_hard_questions` classification [mistake/window→weak, in-window/tested→mastered], ordering by mode [diagnostic weak easy-to-hard → mastered hard-first; beginner globally easy-to-hard], and CLI tests—36 cases) | 🟡 (`difficulty` 1–5 + optional `difficulty_reason` schema fields) | — (deterministic script tool, not a conversation contract, no mock detector) | ❌ | — (does not create long-session behavior) | Honest scope: score is a **heuristic lower bound** from structural signals, not semantic judgment and not per-student (user mistakes/confusions affect selection only and are not written to the shared bank); LLM semantic refinement is a future opt-in | Keep as is (script tests already cover every signal and ordering contract) |
| Structured progress state `study_state.json` (A4) | ✅ (`update_progress` migration/set/add/render/idempotence tests, including persistence of preferences/window) | ✅ (Tier-1 state-schema validation + checkpoint ∈ plan) | 🟡 (mock asserts **only** mistake/confusion state rows in `state_after`; **Tier 0 covers preference/window rows, Tier 2 does not assert them**) | ❌ | 🟡 (replay: state is the sole source of truth, write-after-hand-edit detection `md_write_after_state`, dual-write consistency for mistake/confusion rows) | Real-LLM state updates unverified (opt-in) | Scripted learning event → assert state row and Markdown as generated view |
| Hint / skip / mistake archive | ❌ | 🟡 (progress template contains mistake section) | 🟡 (mock `hint_skip_mistake_archive`: escape route + mistake row written) | ❌ | 🟡 (replay: mistake row added + retained across turns) | Real-LLM escape flow unverified (opt-in) | Two consecutive scripted wrong answers → assert three choices + mistake row |
| `confusion-tracker` behavior | 🟡 (sub-skill exists under `skills/`) | 🟡 (progress template contains confusion section) | 🟡 (mock `confusion_tracking`: one confusion-table row added) | ❌ | 🟡 (replay: confusion row added + retained across turns) | Real-LLM confusion capture unverified (opt-in) | Scripted conceptual question → assert confusion row |
| Notebook persistence receipt (v4 §2.4 persist before summary) | ❌ | — | 🟡 (mock `notebook_persist_ok`: teaching turn must include both `notebook.py add-entry` and a student-visible `notebook/chNN.md#` anchor receipt with matching chapter; chat-only teaching with no persistence is caught) | ❌ | ❌ | Real-LLM persistence behavior unverified (opt-in); actual `notebook/` writes are a state dimension awaiting a drift scenario | Scripted teaching turn → assert receipt command + anchor link |
| Workspace confirmation required (v4 §2.5 workspace location) | ❌ | — | 🟡 (mock `workspace_confirm_ok`: any creation call—`ingest.py --output-dir`/`workspace-register`—must follow a location-confirmation question + affirmative student response; silent creation/create-then-confirm/create-after-rejection all caught) | ❌ | ❌ | Real-LLM first-use guidance unverified (opt-in) | Scripted ingestion request → assert ask before create |
| Quota-friendly artifact routing (`chat` / `visual`) | ❌ | — | 🟡 (mock `artifact_mode_routing`: chat does not render automatically, only explicit visual persists, one-shot PDF does not change state, subscription names do not trigger mode switching, native/browser route selected before preflight) | ❌ | ❌ | Host-native PDF calls by real Agents still require per-runtime testing; deterministic layer verifies command traces and counterexamples | Three scripted state traces + subscription-guess counterexample |
| `study_progress` checkpoint recovery | 🟡 (ingestion rerun does not overwrite progress) | 🟡 (`current_phase ∈ plan`) | 🟡 (mock `checkpoint_recovery`: reads current phase 2 from progress) | ❌ | 🟡 (replay: detects checkpoint reset, resumed vs expected) | Real-LLM resume behavior unverified (opt-in) | Seed progress at phase N → new conversation → assert resume at N |
| Provenance labels 🟢/🟡/⚠️ | ✅ (validator + language tests) | ✅ (`ai_generated` flag required) | 🟡 (mock `provenance_labels`: output contains every canonical label) | 🟡 (indirectly through out-of-scope abstention) | 🟡 (replay: provenance-label fidelity in later explanation turns) | Real-LLM runtime output unverified (opt-in) | Scripted “AI supplement” answer → assert 🟡/⚠️ appears |
| AI-generated answer warning | ✅ (validator rejects an unmarked answer) | ✅ | 🟡 (mock: ⚠️ label + fixture contains `ai_generated` item) | 🟡 | — | Real-LLM runtime output unverified (opt-in) | Same as above |
| Chinese student-visible output | ✅ (language/control-layer tests) | — | ❌ | ❌ | — | Runtime language untested | Scripted run → assert Simplified Chinese output |
| English control layer | ✅ (control-plane tests) | — | — | — | — | Covered | Keep as is |
| Localization boundary | ✅ (localization tests) | — | — | — | — | Covered | Keep as is |
| `web_prompt` fallback | 🟡 (Chinese-first + rules present) | — | ❌ | ❌ | — | Behavior workflow untested | Structure: `web_prompt` contains stepwise flow + provenance labels |
| Path/progress safety | ✅ (many validator + ingestion tests) | ✅ | — | — | — | Covered | Keep as is |
| Workspace validator | ✅ (validator unit tests) | ✅ | — | — | — | Covered | Keep as is |

## How to read this table

- **Many ✅/🟡 entries in the first two columns (Tier 0 / Tier 1)**: structure, schemas, provenance rules, path safety, and ingestion are tested at zero cost.
- **Existing Tier 2 deterministic mock layer (🟡)**: `scenarios.json` contains **17 behavior scenarios**. **16 are asserted in deterministic `--mock`, run in CI, and cost nothing**. One (`lazy_load_best_effort`) is marked `best_effort:true` and reports `SKIP` without an assertion in deterministic mode because it requires a transcript/real LLM. The 16 asserted scenarios comprise the original seven (`quiz_bank`-only / provenance labels / hint-skip-mistake archive / confusion tracking / checkpoint recovery / no-Python fallback / beginner walkthrough upgraded to seven steps), five added by line A (`visual_first_assets` prompt-image gate / `scope_override` override announcement / `teaching_template` seven-step template + source block / `time_budget_no_questions` ≤1-day cadence and explicit no-questions split / `knowledge_window_recheck` out-of-window recheck), one added by A8b (`language_first_ask` combined language question), two v4 red lines (`notebook_persist_ok` teaching persistence receipt / `workspace_confirm_ok` workspace confirmation), and one artifact-resource gate (`artifact_mode_routing`). See [`../behavior_smoke/README.md`](../behavior_smoke/README.en.md) for the inventory.
- **New Tier 4 deterministic replay layer (🟡)**: [`../drift/`](../drift/) covers goal retention, plan adherence, question-invention rate, checkpoint recovery, provenance-label fidelity, progress persistence, and cross-chapter wiki reads across **multi-turn long sessions**. It also covers A6’s `mode_urgent_no_questions` (zero interactive questions after an explicit no-questions request + urgent-opening inference and persistence) and B3’s `window_persist` (long-session knowledge-window persistence through `window_rows_added`/`window_rows_lost`). It replays scripted transcripts plus snapshots in root tests at zero cost.
- **B3: Tier 4 real-Agent route (opt-in)**: `run_drift.py --llm` is no longer a skeleton. It delegates to [`../drift/run_live_smoke.py`](../drift/run_live_smoke.py), which drives real-Agent long sessions with token limits, fail-fast termination, and a fixture sandbox → records T5b logs → converts them to JSONL → grades with the **same** `compute_metrics`/thresholds → records the run. It requires `RUN_SKILL_DRIFT_LLM=1`, incurs real invocation cost, and **never runs in CI**.
- **B1 final-scan principle**: Every fixed defect on line A has a corresponding deterministic test that **turns red**—either a Tier 0 unit test or a negative assertion in a Tier 2 mock detector. This table records each location. Real-LLM behavior remains opt-in and excluded from CI.
- **But 🟡 ≠ ✅**: The deterministic layers (Tier 2 mock and Tier 4 replay) prove only that detectors work on expected artifacts/scripted conversations. They **do not prove that a real LLM Agent will necessarily produce those behaviors**. Real behavior coverage requires the opt-in `--llm` route, which is off by default, unimplemented, and excluded from CI. Lazy loading and compute-before-drawing remain best-effort/uncovered.
- **Most gaps are best filled with deterministic structural assertions** over artifact files and do not require an LLM judge. See [`testing-audit.md`](testing-audit.en.md) §9 and [`../behavior_smoke/README.md`](../behavior_smoke/README.en.md).
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall