ZeKaiNie/universal-examprep-skill / full
universal-exam-cram-coach-full
Verified帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。
benchmark/docs/coverage-matrix.en.md
View on GitHub# Feature × Tier Coverage Matrix
English · [中文](coverage-matrix.md)
> Companion to [`testing-audit.md`](testing-audit.en.md). In one sentence: **many capabilities are tested statically at the structure/instruction layer but have not been executed and verified at the behavior layer.**
**Legend**: ✅ covered 🟡 partial (structure/schema layer, **or** Tier 2 deterministic mock detection—not real LLM behavior) ❌ not covered — not applicable
> **Tier 2 column note**: 🟡 means this PR’s (T2) **deterministic mock detector** covers artifact assertions for the behavior (in CI, zero cost). It **does not mean** real LLM behavior has been verified. Real-Agent smoke is opt-in and excluded from CI; see [`../behavior_smoke/`](../behavior_smoke/).
> **B2 is wired**: `behavior_smoke --llm` single-turn real-Agent smoke is no longer a skeleton. It drives `claude -p` per scenario (or an `--agent-cmd` stub), applies the **same detector function** as `--mock` to the response (the primary positive check for each scenario; mock-only extra negative/variant assertions validate the detector and are not rerun live), writes a transcript, and emits metrics. State/file scenarios that a one-shot `-p` cannot verify are honestly reported as SKIP. This is opt-in, off by default, and excluded from CI; `tests/test_behavior_smoke_live.py` verifies the wiring deterministically with a stub Agent. Every table note saying “real LLM … unverified (opt-in)” refers to this route.
>
> **Tier 4 column note**: 🟡 means PR T4’s **deterministic replay harness** in [`../drift/`](../drift/) covers the behavior across **multi-turn, long-running conversations** by replaying scripted transcripts plus snapshots in root tests at zero cost. It likewise **does not mean** real LLM long-session behavior has been verified. Real-Agent long sessions are opt-in, unimplemented, and excluded from CI.
| Feature | Tier 0 unit/static | Tier 1 workspace validator | Tier 2 behavior smoke | Tier 3 full benchmark | Tier 4 long-horizon drift | Current gap | Cheap metric that could be added |
| :-- | :--: | :--: | :--: | :--: | :--: | :-- | :-- |
| Goal retention | ❌ | — | ❌ | ❌ | 🟡 (replay: rate of topic drift/refusal of original plan) | Real-LLM long sessions unverified (opt-in) | Count topic-drift phrases in scripted long sessions |
| Plan adherence | ❌ | 🟡 (progress `current_phase ∈ plan`) | ❌ | ❌ | 🟡 (replay: unauthorized phase deletion/replacement/addition count) | Real-LLM long sessions unverified (opt-in) | `study_plan` snapshot diff |
| LLM Wiki lazy loading | ❌ (instruction text only) | — | 🟡 (best-effort placeholder `count_wiki_reads`, not asserted in CI) | 🟡 (skill arm uses the wiki but does not test laziness) | 🟡 (replay: cross-chapter wiki reads / unique file count) | A real LLM is required to verify “read only the relevant chapter” | File-read call count in a scripted transcript = one chapter |
| One-command ingestion cold start | ✅ (ingestion end-to-end unit test) | 🟡 (validates artifact structure) | — | ❌ | — | Already covered reasonably well | Keep as is |
| Write-to-disk fallback without Python | ❌ | — | 🟡 (mock `no_python_fallback`: hand-written workspace passes Tier-1 validation) | ❌ | — | A true end-to-end run with Python disabled remains untested | Deterministic: hand-written workspace → `validate_workspace` exits 0 |
| `quiz_bank`-only questions | 🟡 (question-bank schema) | 🟡 (schema) | 🟡 (mock `quiz_bank_only`: question ID ∈ bank + invented IDs detected) | ❌ | 🟡 (replay: invention rate across session / cross-phase questions) | Real-LLM question selection unverified (opt-in) | Scripted question selection → assert every ID ∈ bank |
| Six question types | ✅ (schema accepts all six) | ✅ (schema enforces them) | 🟡 (fixture covers and validates all six) | ❌ (item set contains only factual/definition/numeric) | — | Real-LLM per-type asking/grading unverified | One grading smoke item of each type from a fixed bank |
| Run algorithm before drawing a diagram question | ❌ | 🟡 (`diagram_type` warning) | ❌ (fixture has `render_hint`, but no “compute before drawing” behavior scenario) | ❌ | — | Behavior untested | Transcript computes first, then renders |
| Beginner key-question walkthrough (→ A5 seven-step template) | ❌ | — | 🟡 (mock `zero_basic_key_question`: good case uses all ①–⑦ + source block; legacy two-part form without ①–⑦ fails) | ❌ | — | Real-LLM walkthrough unverified (opt-in) | Scripted “key question” request → assert seven steps + source block |
| Seven-step walkthrough template + per-item source block (A5) | 🟡 (`teaching_template` detector tests: circled numbers / heading boundaries / per-item segmentation / ⑦ provenance) | — | 🟡 (mock `teaching_template`: ①–⑦ complete and ordered, ② before ④, ⑦ reaches wiki/page, per-item source block `题目来源|答案来源|🟢/🟡/⚠️`, double ⚠️ for AI answers, default stop at source block, opt-in closing block; more than ten negative cases caught) | ❌ | ❌ | Real-LLM teaching output unverified (opt-in) | Scripted teaching → assert seven steps + source block + ⚠️ |
| Visual-question prompt-image gate (P0-V1 / A1) | ✅ (`show_question_assets`/index-script unit tests) | ✅ (validator rejects a workspace where a visually required item lacks a prompt-side asset) | 🟡 (mock `visual_first_assets`: prompt image first + 11 negative cases—answer image first/answer leak/unlabeled/late/unsafe path/path printed only—all caught) | ❌ | ❌ | Real-LLM rendering order unverified (opt-in) | Scripted visual question → assert prompt image precedes explanation/answer image |
| Scope filtering + out-of-scope override announcement (A2) | 🟡 (`select_questions` selection by `source_type` unit test) | 🟡 (schema: `source_type` label) | 🟡 (mock `scope_override`: verbatim `⚠️ 临时覆盖…范围偏好` must precede the first out-of-scope question; negative case caught) | ❌ | ❌ | Real-LLM scope switching unverified (opt-in) | Scripted out-of-scope question → assert override announcement precedes it |
| Learning mode × time budget (A6) | ✅ (`update_progress` set/normalization/legacy-four-mode migration/unknown-value preservation tests; urgent-alias normalization; explicit `no_questions` cap) | 🟡 (state schema: mode/time_budget/preferences) | 🟡 (mock `time_budget_no_questions`: allows a real-bank checkpoint under ordinary ≤1 day, catches clarification/preference/reflection questions; zero interactive questions under explicit `no_questions`) | ❌ | 🟡 (replay `mode_urgent_no_questions`: explicit no-questions scenario requires `urgent_mode_questions`=0 + `urgent_mode_persisted`=1) | Real-LLM cadence by time tier unverified (opt-in) | Two scripted conversations: ordinary ≤1 day allows bank checkpoint; explicit no-questions has zero interactive questions + persistence |
| Combined first language question + dispatch (A8b) | 🟡 (`tests/test_study_state.py` language normalization + `tests/test_language_purity.py` bilingual composition + EN vocabulary/purity + `tests/test_language_policy.py` dispatch pins) | ❌ | 🟡 (mock `language_first_ask`: first question contains three language rows; accepts three `set` flags in any order, catches noncanonical values; urgent defaults + never infers bilingual) | ❌ | ❌ | Long-session consistency of EN output untested | Add an EN variant to a drift scenario (future item, independent of the delivered A8c EN entry surfaces) |
| Four-part cheatsheet layout (phase 6 C1: 必背→例题→例题解答→要点解释, examples visual-first) | 🟡 (`tests/test_language_policy.py` section pins + control-layer section pins + localization pins) | ❌ | ❌ | ❌ | ❌ | Static contract only; no output behavior detector | Low priority: register if a cheatsheet smoke scenario is added later |
| Knowledge window (A6, 3–7 days/>7 days) | ✅ (`window-add`/`window-set-status` tests: chapter-scoped lookup, multi-chapter ambiguity fail-loud, idempotent backfill, lossless init round trip) | 🟡 (state schema: knowledge_window object array) | 🟡 (mock `knowledge_window_recheck`: out-of-window knowledge must be genuinely rechecked; >7 days must test with a question; negation/default closing does not count—**wording only, no state row written**) | ❌ | 🟡 (B3 replay `window_persist`: long sessions register `window_rows_added`≥2 while teaching and change status without deleting rows, `window_rows_lost`=0; silent row loss fails; malformed row without `point` fails loud) | Real-LLM knowledge-window transitions unverified (opt-in) | Scripted out-of-window point → assert recall question or test |
| Difficulty scoring + hard-question selection (A7) | ✅ (`score_difficulty` tests for every signal dimension [cross-knowledge/structure/needs figure/multiple pages/chapter position/question type] + atomic rewrite/idempotence/dry-run/symlink rejection; `select_hard_questions` classification [mistake/window→weak, in-window/tested→mastered], ordering by mode [diagnostic weak easy-to-hard → mastered hard-first; beginner globally easy-to-hard], and CLI tests—36 cases) | 🟡 (`difficulty` 1–5 + optional `difficulty_reason` schema fields) | — (deterministic script tool, not a conversation contract, no mock detector) | ❌ | — (does not create long-session behavior) | Honest scope: score is a **heuristic lower bound** from structural signals, not semantic judgment and not per-student (user mistakes/confusions affect selection only and are not written to the shared bank); LLM semantic refinement is a future opt-in | Keep as is (script tests already cover every signal and ordering contract) |
| Structured progress state `study_state.json` (A4) | ✅ (`update_progress` migration/set/add/render/idempotence tests, including persistence of preferences/window) | ✅ (Tier-1 state-schema validation + checkpoint ∈ plan) | 🟡 (mock asserts **only** mistake/confusion state rows in `state_after`; **Tier 0 covers preference/window rows, Tier 2 does not assert them**) | ❌ | 🟡 (replay: state is the sole source of truth, write-after-hand-edit detection `md_write_after_state`, dual-write consistency for mistake/confusion rows) | Real-LLM state updates unverified (opt-in) | Scripted learning event → assert state row and Markdown as generated view |
| Hint / skip / mistake archive | ❌ | 🟡 (progress template contains mistake section) | 🟡 (mock `hint_skip_mistake_archive`: escape route + mistake row written) | ❌ | 🟡 (replay: mistake row added + retained across turns) | Real-LLM escape flow unverified (opt-in) | Two consecutive scripted wrong answers → assert three choices + mistake row |
| `confusion-tracker` behavior | 🟡 (sub-skill exists under `skills/`) | 🟡 (progress template contains confusion section) | 🟡 (mock `confusion_tracking`: one confusion-table row added) | ❌ | 🟡 (replay: confusion row added + retained across turns) | Real-LLM confusion capture unverified (opt-in) | Scripted conceptual question → assert confusion row |
| Notebook persistence receipt (v4 §2.4 persist before summary) | ❌ | — | 🟡 (mock `notebook_persist_ok`: teaching turn must include both `notebook.py add-entry` and a student-visible `notebook/chNN.md#` anchor receipt with matching chapter; chat-only teaching with no persistence is caught) | ❌ | ❌ | Real-LLM persistence behavior unverified (opt-in); actual `notebook/` writes are a state dimension awaiting a drift scenario | Scripted teaching turn → assert receipt command + anchor link |
| Workspace confirmation required (v4 §2.5 workspace location) | ❌ | — | 🟡 (mock `workspace_confirm_ok`: any creation call—`ingest.py --output-dir`/`workspace-register`—must follow a location-confirmation question + affirmative student response; silent creation/create-then-confirm/create-after-rejection all caught) | ❌ | ❌ | Real-LLM first-use guidance unverified (opt-in) | Scripted ingestion request → assert ask before create |
| Quota-friendly artifact routing (`chat` / `visual`) | ❌ | — | 🟡 (mock `artifact_mode_routing`: chat does not render automatically, only explicit visual persists, one-shot PDF does not change state, subscription names do not trigger mode switching, native/browser route selected before preflight) | ❌ | ❌ | Host-native PDF calls by real Agents still require per-runtime testing; deterministic layer verifies command traces and counterexamples | Three scripted state traces + subscription-guess counterexample |
| `study_progress` checkpoint recovery | 🟡 (ingestion rerun does not overwrite progress) | 🟡 (`current_phase ∈ plan`) | 🟡 (mock `checkpoint_recovery`: reads current phase 2 from progress) | ❌ | 🟡 (replay: detects checkpoint reset, resumed vs expected) | Real-LLM resume behavior unverified (opt-in) | Seed progress at phase N → new conversation → assert resume at N |
| Provenance labels 🟢/🟡/⚠️ | ✅ (validator + language tests) | ✅ (`ai_generated` flag required) | 🟡 (mock `provenance_labels`: output contains every canonical label) | 🟡 (indirectly through out-of-scope abstention) | 🟡 (replay: provenance-label fidelity in later explanation turns) | Real-LLM runtime output unverified (opt-in) | Scripted “AI supplement” answer → assert 🟡/⚠️ appears |
| AI-generated answer warning | ✅ (validator rejects an unmarked answer) | ✅ | 🟡 (mock: ⚠️ label + fixture contains `ai_generated` item) | 🟡 | — | Real-LLM runtime output unverified (opt-in) | Same as above |
| Chinese student-visible output | ✅ (language/control-layer tests) | — | ❌ | ❌ | — | Runtime language untested | Scripted run → assert Simplified Chinese output |
| English control layer | ✅ (control-plane tests) | — | — | — | — | Covered | Keep as is |
| Localization boundary | ✅ (localization tests) | — | — | — | — | Covered | Keep as is |
| `web_prompt` fallback | 🟡 (Chinese-first + rules present) | — | ❌ | ❌ | — | Behavior workflow untested | Structure: `web_prompt` contains stepwise flow + provenance labels |
| Path/progress safety | ✅ (many validator + ingestion tests) | ✅ | — | — | — | Covered | Keep as is |
| Workspace validator | ✅ (validator unit tests) | ✅ | — | — | — | Covered | Keep as is |
## How to read this table
- **Many ✅/🟡 entries in the first two columns (Tier 0 / Tier 1)**: structure, schemas, provenance rules, path safety, and ingestion are tested at zero cost.
- **Existing Tier 2 deterministic mock layer (🟡)**: `scenarios.json` contains **17 behavior scenarios**. **16 are asserted in deterministic `--mock`, run in CI, and cost nothing**. One (`lazy_load_best_effort`) is marked `best_effort:true` and reports `SKIP` without an assertion in deterministic mode because it requires a transcript/real LLM. The 16 asserted scenarios comprise the original seven (`quiz_bank`-only / provenance labels / hint-skip-mistake archive / confusion tracking / checkpoint recovery / no-Python fallback / beginner walkthrough upgraded to seven steps), five added by line A (`visual_first_assets` prompt-image gate / `scope_override` override announcement / `teaching_template` seven-step template + source block / `time_budget_no_questions` ≤1-day cadence and explicit no-questions split / `knowledge_window_recheck` out-of-window recheck), one added by A8b (`language_first_ask` combined language question), two v4 red lines (`notebook_persist_ok` teaching persistence receipt / `workspace_confirm_ok` workspace confirmation), and one artifact-resource gate (`artifact_mode_routing`). See [`../behavior_smoke/README.md`](../behavior_smoke/README.en.md) for the inventory.
- **New Tier 4 deterministic replay layer (🟡)**: [`../drift/`](../drift/) covers goal retention, plan adherence, question-invention rate, checkpoint recovery, provenance-label fidelity, progress persistence, and cross-chapter wiki reads across **multi-turn long sessions**. It also covers A6’s `mode_urgent_no_questions` (zero interactive questions after an explicit no-questions request + urgent-opening inference and persistence) and B3’s `window_persist` (long-session knowledge-window persistence through `window_rows_added`/`window_rows_lost`). It replays scripted transcripts plus snapshots in root tests at zero cost.
- **B3: Tier 4 real-Agent route (opt-in)**: `run_drift.py --llm` is no longer a skeleton. It delegates to [`../drift/run_live_smoke.py`](../drift/run_live_smoke.py), which drives real-Agent long sessions with token limits, fail-fast termination, and a fixture sandbox → records T5b logs → converts them to JSONL → grades with the **same** `compute_metrics`/thresholds → records the run. It requires `RUN_SKILL_DRIFT_LLM=1`, incurs real invocation cost, and **never runs in CI**.
- **B1 final-scan principle**: Every fixed defect on line A has a corresponding deterministic test that **turns red**—either a Tier 0 unit test or a negative assertion in a Tier 2 mock detector. This table records each location. Real-LLM behavior remains opt-in and excluded from CI.
- **But 🟡 ≠ ✅**: The deterministic layers (Tier 2 mock and Tier 4 replay) prove only that detectors work on expected artifacts/scripted conversations. They **do not prove that a real LLM Agent will necessarily produce those behaviors**. Real behavior coverage requires the opt-in `--llm` route, which is off by default, unimplemented, and excluded from CI. Lazy loading and compute-before-drawing remain best-effort/uncovered.
- **Most gaps are best filled with deterministic structural assertions** over artifact files and do not require an LLM judge. See [`testing-audit.md`](testing-audit.en.md) §9 and [`../behavior_smoke/README.md`](../behavior_smoke/README.en.md).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |