Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/behavior_smoke/README.en.md

View on GitHub
# Tier 2 — Behavioral Smoke

English · [中文](README.md)

This tests the skill as a **tutoring workflow**, rather than testing only static files (Tier 0/1) or grounded Q&A (Tier 3 benchmark).
Definitions are in [`../docs/test_tiers.en.md`](../docs/test_tiers.en.md); the current-state audit is in [`../docs/testing-audit.en.md`](../docs/testing-audit.en.md).

## Two paths

| Path | Command | Cost | In CI? |
| :-- | :-- | :-- | :--: |
| **Default (deterministic)** | `--mock` / `--check-fixture` | $0, pure stdlib, no network, no LLM, no API key | ✅ |
| **Optional (real agent)** | `--llm` (requires `RUN_SKILL_BEHAVIOR_LLM=1`) | Runs `claude -p` (subscription, no API key needed) | ❌ Disabled by default |

```bash
# Default: zero cost, can run in CI
python benchmark/behavior_smoke/run_behavior_smoke.py --check-fixture   # validate the mini-course workspace
python benchmark/behavior_smoke/run_behavior_smoke.py --mock            # run the deterministic detectors on mock outputs

# Optional: real-agent smoke (both the env var and the flag are required; otherwise it refuses to run)
RUN_SKILL_BEHAVIOR_LLM=1 python benchmark/behavior_smoke/run_behavior_smoke.py --llm
```

With no arguments it only prints help and **never** calls an LLM.

## What it covers

The self-authored small workspace [`fixtures/mini_course/`](fixtures/mini_course) (**no copyrighted content**, general CS knowledge) covers all 6 question types
and passes `scripts/validate_workspace.py`. Each scenario ([`scenarios.json`](scenarios.json)) has one **deterministic detector**:

| Scenario | Behavior claim | Default (deterministic) check |
| :-- | :-- | :-- |
| `quiz_bank_only` | Quizzes use only the question bank; no improvised questions | Every question ID in the mock output must be in `quiz_bank.json`; the negative example (made-up question IDs) must fail |
| `provenance_labels` | Distinguishes sources and uses the canonical labels | The mock output contains all of the 🟢/🟡/⚠️ canonical labels |
| `hint_skip_mistake_archive` | After two wrong answers in a row, offers a hint/skip/archive | The mock output contains the escape hatch + the mock progress has a mistake row |
| `confusion_tracking` | "Why"-type questions are written to progress | A new row appears in the confusion section of the mock progress |
| `checkpoint_recovery` | Resumes from the current phase instead of restarting | Reads current phase = 2 from progress, and the resume message points to phase 2 |
| `no_python_fallback` | Hand-written output without Python is still complete | The hand-written workspace passes Tier-1 validation |
| `zero_basic_key_question` | A zero-background walkthrough has structured sections | The mock output contains 考点拆解 ("breakdown of the tested point") (or 这题在问什么, "what this question asks") + 标准答题步骤 ("standard answer steps") (or 逐步演算, "step-by-step work"); 易错点 ("common mistakes") / 3分钟速记 ("3-minute memo") are optional closing blocks and are no longer required |
| `teaching_template` | A5 seven-step walkthrough template + a source block for every question | ①-⑦ all present and in order (② before ④), and ⑦ points to the chapter/wiki; the source line has 题目来源 ("question source") | 答案来源 ("answer source") | canonical label; for an AI answer, ⚠️ appears in the source line and in the answer block heading; by default the output stops at the source block, and unrequested closing blocks are caught (exempt if the student asked for them); all 7 negative examples are caught |
| `visual_first_assets` | Visual questions show the question-side asset first | The mock output must first show a real local fixture image carrying the `题面图 / question-side asset` label; the negative examples (answer image shown first / answer image or text leaked before the question / unlabeled answer image / text before the image / image inserted after the question / image arriving late after `问题:` ("Question:") / unsafe or missing path / path printed only) must fail |
| `scope_override` | Quizzing outside the chosen scope must be announced first (A2) | The mock output shows, verbatim and **before** the first question, 「⚠️ 临时覆盖你的 <范围> 范围偏好」 ("⚠️ Temporarily overriding your <scope> scope preference"); the negative examples (announced only after the question / not announced) must fail |
| `language_first_ask` | The first question asks mode × time × language together, once (A6/A8b), with the language line listing all three language options; an urgent opening is inferred silently | Mock good example = language line with all three options + one set call with all three flags; the negative example missing the language line is caught; urgent variant = zero opening clarification/preference questions + `--language` ∈ canonical values, and the urgent negative example that ends with a question is caught |
| `time_budget_no_questions` | ≤1天 (≤1 day) pacing + explicit `no_questions` (A6) | Ordinary ≤1天 allows bank checkpoints that carry a real `[#id]` and a matching prompt; clarifying/preference/reflective follow-up questions must fail. When the scenario prompt explicitly says “别问我问题” ("don't ask me questions"), every interactive question (checkpoints included) is forbidden; rhetorical questions the tutor answers itself are not falsely flagged |
| `knowledge_window_recheck` | Knowledge points outside the window must really be rechecked (A6) | For 3-7天 (3–7 days), the good example may either ask back or actually test, and the negative example is still caught by default; for >7天 (`require_test`), only an actual quiz counts, so the bad example that only asks verbally 「还记得吗」 ("do you still remember?") is caught; negative phrasings (不问/不实测/我就当你会了, "won't ask / won't test / I'll just assume you know it") do not count as a recheck, and negated safety statements (不会默认你会, "I won't assume you know it") are not falsely flagged |
| `artifact_mode_routing` | Quota-friendly `chat` / `visual` artifact gate | Verified on the mock command trace: finishing a chapter in `chat` does not auto-render; `visual` is persisted only after an explicit standing choice; a subscription name alone cannot trigger a switch; after a one-shot PDF the mode is still `chat`; for a PDF, a native/browser backend must be chosen before this chapter's preflight and generation. The negative examples catch, respectively, an automatic PDF, guessing the subscription, and a one-shot request polluting the standing state |
| `notebook_persist_ok` | A teaching turn "writes to disk first, then summarizes" (v4 §2.4 red line) | The mock good example contains both the `notebook.py … add-entry` write command (also recognized inside a code span) and a student-visible `notebook/chNN.md#锚点` (anchor) receipt (the zh canonical form looks like `完整解答:notebook/ch02.md#q13`, "full solution: …"; the receipt's chapter number must match the command's `--chapter` with zero padding); the negative example that explains everything only in chat, with zero write receipts, must fail |
| `workspace_confirm_ok` | Creating a workspace requires confirmation; silently creating one = a violation (v4 §2.5 red line) | In the mock good example, **before** the first create call (`ingest.py --output-dir` / `workspace-register`), the tutor asks to confirm the location and the student affirms the target path; the negative example that silently creates the workspace with `ingest.py --output-dir` right at the start must fail; asking without waiting for the answer, creating first and seeking approval afterward, or creating after the student declined all fail as well |
| `lazy_load_best_effort` | Reads only the current chapter | **best-effort**: skipped in deterministic mode; a real check needs a transcript/LLM |

## What is best-effort / not covered

- **`lazy_load`** can verify "read only the current chapter" only with a real tool-call transcript (or a real `--llm` run);
  deterministic mode provides only a placeholder detector, `count_wiki_reads`, which is **not asserted in CI**.
- **Deterministic mode only proves that the detector logic holds on mock outputs**. It does **not** prove that a real agent will produce these behaviors.
  Covering real LLM behavior requires running the optional `--llm` path (off by default, not in CI).
- **The deterministic detectors are smoke heuristics, not semantic graders**: they use structure / question IDs / prompt matching / chapter scope / negation words
  and similar signals to catch **common** fabrications and misjudgments (made-up questions, a valid question ID attached to a made-up prompt, unnumbered questions, negated escape hatches, empty-state placeholder rows,
  drifting to the wrong chapter, listing a label legend without labeling the answer, and so on), but they **cannot exhaust** every rewording an arbitrary LLM might produce. Real semantic judgment is left to the opt-in
  `--llm` path and a future LLM judge (Tier 3/4). That is also why this sits under "behavioral smoke" rather than "behavioral grading".

## Boundaries (what this is not)

- This is **not** a full benchmark and does **not replace** Tier 3 (full matrix) / Tier 4 (long-horizon drift).
- The default path does **not** run a model, does **not** use the network, does **not** read an API key, and does **not** cost anything.
- `--llm` is wired up (B2): it drives a real agent for a single turn (`claude -p`, or any/stub command passed via `--agent-cmd`), applies the **same** detectors as `--mock` to every scenario that can be verified from the reply, writes the transcript, and outputs metrics; opt-in, off by default, not in CI. The wiring is verified deterministically by `tests/test_behavior_smoke_live.py` with a stub agent (state/file scenarios cannot be verified with a one-shot `-p` and are honestly reported as SKIP).
- Outputs go to `results/` (gitignored), and fixtures are copied to a temporary directory before anything modifies them.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 323kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 323kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 323kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 323kUpdated Claude CodeCodexVerifiedInstall