Skip to content

ZeKaiNie/universal-examprep-skill / full

universal-exam-cram-coach-full

Verified

帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。

FileKindSize
.gitattributesother1.5 KB
.github/workflows/ci.ymlconfig1.9 KB
.gitignoreother659 B
AGENTS.mdmarkdown38.8 KB
CHANGELOG.en.mdmarkdown25.5 KB
CHANGELOG.mdmarkdown20.5 KB
CONTRIBUTING.mdmarkdown5.1 KB
CONTRIBUTING.zh.mdmarkdown4.9 KB
LICENSEother1.0 KB
README.mdmarkdown24.5 KB
README.zh.mdmarkdown22.8 KB
SKILL.mdskill md10.4 KB
assets/exam-panic.pngbinary934.2 KB
benchmark/.gitignoreother1.2 KB
benchmark/README.en.mdmarkdown12.7 KB
benchmark/README.mdmarkdown10.8 KB
benchmark/REPORT.en.mdmarkdown7.1 KB
benchmark/REPORT.mdmarkdown6.4 KB
benchmark/aggregate_matrix.pyscript16.3 KB
benchmark/behavior_smoke/README.en.mdmarkdown9.9 KB
benchmark/behavior_smoke/README.mdmarkdown7.4 KB
benchmark/behavior_smoke/fixtures/mini_course/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course/references/quiz_bank.jsonconfig4.5 KB
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/fixtures/mini_course/study_state.jsonconfig390 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/assets/venn_prompt.svgother677 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/quiz_bank.jsonconfig3.2 KB
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch1_stack_queue.mdmarkdown447 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/references/wiki/ch2_trees.mdmarkdown493 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_plan.mdmarkdown550 B
benchmark/behavior_smoke/fixtures/mini_course_nostate/study_progress.mdmarkdown429 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_auto_pdf_bad.txtmarkdown433 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_good.txtmarkdown462 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_chat_not_persisted_bad.txtmarkdown186 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_good.txtmarkdown490 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_persist_bad.txtmarkdown605 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_one_shot_probe_error_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_native_good.txtmarkdown495 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_preflight_failed_bad.txtmarkdown487 B
benchmark/behavior_smoke/mock/sample_outputs/artifact_mode_visual_subscription_guess_bad.txtmarkdown507 B
benchmark/behavior_smoke/mock/sample_outputs/hint_skip_offer.txtmarkdown349 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_good.txtmarkdown461 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_missing_line.txtmarkdown270 B
benchmark/behavior_smoke/mock/sample_outputs/language_ask_unglossed.txtmarkdown211 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_good.txtmarkdown216 B
benchmark/behavior_smoke/mock/sample_outputs/language_persist_noncanonical.txtmarkdown157 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_explicit.txtmarkdown372 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_bilingual_negated.txtmarkdown306 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_question.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_silent.txtmarkdown364 B
benchmark/behavior_smoke/mock/sample_outputs/language_urgent_wrong_language.txtmarkdown262 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_chat_only.txtmarkdown952 B
benchmark/behavior_smoke/mock/sample_outputs/notebook_persist_good.txtmarkdown744 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_confusion.mdmarkdown395 B
benchmark/behavior_smoke/mock/sample_outputs/progress_after_mistake.mdmarkdown403 B
benchmark/behavior_smoke/mock/sample_outputs/provenance_answer.txtmarkdown492 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_good.txtmarkdown351 B
benchmark/behavior_smoke/mock/sample_outputs/quiz_output_invented.txtmarkdown317 B
benchmark/behavior_smoke/mock/sample_outputs/resume_message.txtmarkdown224 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_good.txtmarkdown258 B
benchmark/behavior_smoke/mock/sample_outputs/scope_override_missing.txtmarkdown127 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_ai_answer_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_formula_first.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_good.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_liberal_good.txtmarkdown1.4 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_missing_warn.txtmarkdown608 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_no_source.txtmarkdown1.1 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_optin_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_skip_ask.txtmarkdown948 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unlabeled_source.txtmarkdown517 B
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_unsolicited_closers.txtmarkdown1.5 KB
benchmark/behavior_smoke/mock/sample_outputs/teaching_template_warn_title.txtmarkdown581 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bad.txtmarkdown340 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_bank_checkpoint.txtmarkdown341 B
benchmark/behavior_smoke/mock/sample_outputs/time_budget_1day_good.txtmarkdown695 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_after_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_before_prompt.txtmarkdown264 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_side_first.txtmarkdown261 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_answer_text_before_prompt.txtmarkdown243 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_good.txtmarkdown232 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_missing_asset.txtmarkdown182 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_path_only.txtmarkdown163 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_prose_before_image.txtmarkdown234 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_question_label_late_asset.txtmarkdown250 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unlabeled_solution_before_prompt.txtmarkdown233 B
benchmark/behavior_smoke/mock/sample_outputs/visual_first_unsafe_url.txtmarkdown178 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_bad.txtmarkdown346 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_good.txtmarkdown665 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_recall_only_bad.txtmarkdown333 B
benchmark/behavior_smoke/mock/sample_outputs/window_recheck_test_good.txtmarkdown580 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_good.txtmarkdown797 B
benchmark/behavior_smoke/mock/sample_outputs/workspace_confirm_silent.txtmarkdown453 B
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_explain.txtmarkdown1.2 KB
benchmark/behavior_smoke/mock/sample_outputs/zero_basic_legacy_only.txtmarkdown442 B
benchmark/behavior_smoke/mock/sample_transcript.jsonlother517 B
benchmark/behavior_smoke/mock/state_after_confusion.jsonconfig203 B
benchmark/behavior_smoke/mock/state_after_mistake.jsonconfig215 B
benchmark/behavior_smoke/run_behavior_smoke.pyscript97.2 KB
benchmark/behavior_smoke/scenarios.jsonconfig12.7 KB
benchmark/calibrate.pyscript7.3 KB
benchmark/calibrate_matrix.pyscript20.0 KB
benchmark/calibration/README.en.mdmarkdown3.2 KB
benchmark/calibration/README.mdmarkdown2.6 KB
benchmark/calibration/kappa_n24_summary.jsonconfig1.5 KB
benchmark/config.example.jsonconfig431 B
benchmark/config.matrix.example.jsonconfig1.1 KB
benchmark/docs/coverage-matrix.en.mdmarkdown17.6 KB
benchmark/docs/coverage-matrix.mdmarkdown15.2 KB
benchmark/docs/img/hard_algo_correct_en.svgother2.7 KB
benchmark/docs/img/hard_algo_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_algo_hallu_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_correct_zh.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_en.svgother2.7 KB
benchmark/docs/img/hard_psyc_hallu_zh.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_en.svgother2.7 KB
benchmark/docs/img/oos_algo_abst_zh.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_en.svgother2.7 KB
benchmark/docs/img/oos_psyc_abst_zh.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_en.svgother2.7 KB
benchmark/docs/img/realexam_algo_correct_zh.svgother2.7 KB
benchmark/docs/judge-calibration.en.mdmarkdown13.1 KB
benchmark/docs/judge-calibration.mdmarkdown9.9 KB
benchmark/docs/materials_sources.en.mdmarkdown3.2 KB
benchmark/docs/materials_sources.mdmarkdown2.7 KB
benchmark/docs/matrix_pipeline.en.mdmarkdown9.5 KB
benchmark/docs/matrix_pipeline.mdmarkdown8.4 KB
benchmark/docs/related_benchmarks.en.mdmarkdown5.5 KB
benchmark/docs/related_benchmarks.mdmarkdown4.6 KB
benchmark/docs/running-real-runs.en.mdmarkdown10.0 KB
benchmark/docs/running-real-runs.mdmarkdown8.0 KB
benchmark/docs/test_tiers.en.mdmarkdown7.7 KB
benchmark/docs/test_tiers.mdmarkdown6.8 KB
benchmark/docs/testing-audit.en.mdmarkdown17.2 KB
benchmark/docs/testing-audit.mdmarkdown14.6 KB
benchmark/docs/testing-walkthrough.en.mdmarkdown23.5 KB
benchmark/docs/测试流程详解.mdmarkdown19.0 KB
benchmark/drift/README.en.mdmarkdown14.6 KB
benchmark/drift/README.mdmarkdown12.3 KB
benchmark/drift/convert_session_log.pyscript12.9 KB
benchmark/drift/docs/live_agent_pilot.mdmarkdown6.6 KB
benchmark/drift/fixtures/live_logs/good_session.expected.jsonlother1.5 KB
benchmark/drift/fixtures/live_logs/good_session.mdmarkdown1.6 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.jsonlother2.2 KB
benchmark/drift/fixtures/live_logs/live_smoke_golden.mdmarkdown2.3 KB
benchmark/drift/fixtures/mini_course_long/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/references/quiz_bank.jsonconfig1.3 KB
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch1_stack_queue.mdmarkdown541 B
benchmark/drift/fixtures/mini_course_long_state/references/wiki/ch2_trees.mdmarkdown413 B
benchmark/drift/fixtures/mini_course_long_state/study_plan.mdmarkdown426 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.bad.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.final.good.mdmarkdown272 B
benchmark/drift/fixtures/mini_course_long_state/study_progress.initial.mdmarkdown151 B
benchmark/drift/fixtures/mini_course_long_state/study_state.jsonconfig145 B
benchmark/drift/run_drift.pyscript79.7 KB
benchmark/drift/run_live_smoke.pyscript26.4 KB
benchmark/drift/scenarios/live_smoke_basic.jsonconfig681 B
benchmark/drift/scenarios/long_session_basic.jsonconfig793 B
benchmark/drift/scenarios/long_session_state.jsonconfig994 B
benchmark/drift/scenarios/mode_urgent_no_questions.jsonconfig1.1 KB
benchmark/drift/scenarios/window_persist.jsonconfig1.0 KB
benchmark/drift/templates/live_session_template.mdmarkdown1.3 KB
benchmark/drift/templates/live_smoke_turns.jsonconfig1.6 KB
benchmark/drift/transcripts/bad_checkpoint_reset.jsonlother877 B
benchmark/drift/transcripts/bad_goal_drift.jsonlother226 B
benchmark/drift/transcripts/bad_plan_drift.jsonlother729 B
benchmark/drift/transcripts/bad_progress_loss.jsonlother1013 B
benchmark/drift/transcripts/bad_provenance_drift.jsonlother583 B
benchmark/drift/transcripts/bad_quiz_invention.jsonlother590 B
benchmark/drift/transcripts/bad_urgent_1day_questions.jsonlother840 B
benchmark/drift/transcripts/bad_wiki_overread.jsonlother430 B
benchmark/drift/transcripts/good_session.jsonlother5.5 KB
benchmark/drift/transcripts/good_session_state.jsonlother7.0 KB
benchmark/drift/transcripts/good_session_urgent_1day.jsonlother2.8 KB
benchmark/drift/transcripts/window_persist_session.jsonlother7.3 KB
benchmark/fixtures/mini_course_matrix/.gitignoreother250 B
benchmark/fixtures/mini_course_matrix/config.jsonconfig394 B
benchmark/fixtures/mini_course_matrix/items.jsonlother1.6 KB
benchmark/fixtures/mini_course_matrix/materials/_combined.txtmarkdown502 B
benchmark/fixtures/mini_course_matrix/raw_ws/lecture01.mdmarkdown379 B
benchmark/fixtures/mini_course_matrix/skill_ws/references/wiki/ch01_minios.mdmarkdown485 B
benchmark/gen.pyscript13.0 KB
benchmark/items/README.en.mdmarkdown3.3 KB
benchmark/items/README.mdmarkdown2.6 KB
benchmark/items/items.example.jsonlother2.0 KB
benchmark/items/items_algo_full_q.jsonlother13.5 KB
benchmark/items/items_pilot_6006.jsonlother4.5 KB
benchmark/items/items_pilot_psyc110.jsonlother4.5 KB
benchmark/items/items_psyc_full_q.jsonlother7.5 KB

benchmark/drift/README.en.md

View on GitHub
# Tier 4 · Long-horizon drift harness (deterministic replay)

English · [中文](README.md)

This is the **first working implementation** of Tier 4, "long-horizon drift": a **deterministic, zero-cost, standard-library-only**
**replay** framework that measures whether a multi-turn tutoring session **stays steadily on the study goal** or **slowly drifts off course** as the turns add up.

> ⚠️ **It replays scripted transcripts; it does not run a real agent.** So it measures "whether a **recorded** session drifted,"
> **not** "whether a **live** model will drift." Real-LLM long sessions remain a **future / opt-in** path (see the end of this file); this directory does not connect to one, pay for calls, or go online.

## How T2 differs from T4

| | Tier 2 behavior smoke | Tier 4 long-horizon drift (this directory) |
| :-- | :-- | :-- |
| Time span | **Single-turn / small-sample** behavior | **Long multi-turn sessions** (a dozen to several dozen turns) |
| Focus | Whether a single output complies (source labels, quizzing only from the question bank…) | Whether the session, **over time**, **keeps the goal / does not change the plan on its own / does not get messier and messier** |
| Relationship | A cheap early warning for Tier 4 | Uses the same kind of deterministic detectors, but accumulates them across the **whole session** |

Both layers are **deterministic mock/replay**: they prove "the detectors hold for the **expected artifacts**," and do **not** prove that a real LLM will actually behave this way.

## Inputs

- **scenario** (`scenarios/*.json`): `name` / `fixture` (workspace snapshot) / `transcript` (the session replayed by default) / `thresholds`
  (table below) / optional `goal_markers`, `unrelated_goal_phrases`.
- **transcript** (`transcripts/*.jsonl`): one line per turn; every field is optional:
  ```json
  {"turn": 3, "user": "从阶段1考我", "assistant": "题目 [#stack_lifo_1] …", "kind": "quiz",
   "phase_context": 1,
   "events": [{"type": "read_file", "path": "references/wiki/ch1_stack_queue.md"},
              {"type": "write_file", "path": "study_progress.md"}],
   "files_after": {"study_progress.md": "…整份快照…"},
   "tokens_in": 900, "tokens_out": 120, "cost_usd": 0.001}
  ```
  `files_after` carries the snapshot of the workspace files (progress / plan) **after** that turn; the drift metrics are computed deterministically from **the differences between adjacent snapshots**.
- **Workspace snapshot parsing** accepts both **this harness's simple format** (`## 阶段1` ("Phase 1"), `当前阶段:1` ("current phase: 1"), `- ` lines) **and the real `scripts/ingest.py` template format** (`| **阶段 1** | … |` tables / `- [ ] **阶段 1**` checkboxes, `* **当前进行阶段**:阶段 1:…` ("phase in progress: Phase 1: …"), Markdown table rows for mistakes/confusions), so it can run self-authored fixtures and also replay sessions recorded in real workspaces.
- **fixture** (`fixtures/mini_course_long/`, self-authored, not copyrighted): `study_plan.md` (a fixed phase sequence),
  `study_progress.initial.md` (the starting phase), `study_progress.final.{good,bad}.md` (good/bad reference end states),
  `references/wiki/ch{1,2}_*.md`, `references/quiz_bank.json` (stable question IDs + `phase`).

## Metrics and thresholds

| Metric | Meaning | Matching threshold |
| :-- | :-- | :-- |
| `goal_retention` | Share of assistant turns that neither go off topic nor reject the original plan | `goal_retention_min` |
| `plan_mutations` / `plan_adherence` | Number of times a phase was deleted/replaced/added without the user's consent | `plan_mutations_max` |
| `invention_rate` (`quiz_items`/`bank_backed`/`invented`/`untagged_questions`/`wrong_phase_quiz`) | Whether every quiz question comes from the question bank, belongs to the right phase, and carries a `[#id]` | `quiz_invention_rate_max` / `untagged_questions_max` / `wrong_phase_quiz_max` |
| `reset_detected` (`resumed_phase`/`expected_phase`) | Whether resuming from a checkpoint continues from the current phase (instead of falling back to Phase 1) | `checkpoint_reset_max` |
| `provenance_fidelity` | Whether later **explanation turns** still carry the 🟢/🟡/⚠️ content labels (using the same check as T2; a legend does not count) | `provenance_fidelity_min` |
| `mistake_rows_added` / `confusion_rows_added` / `progress_rows_lost` | Whether mistakes/confusions get recorded and are **not silently deleted** later | `progress_rows_lost_max` |
| `wiki_reads` / `unique_wiki_files` / `overread_flag` | Whether only the current phase's chapter wiki is read (lazy loading) | `wiki_unique_files_max` / `overread_max` |
| `cost.*` (optional) | `total_tokens_in/out`, `total_cost_usd`, `context_growth_ratio` (tokens_in of the last turn / the first turn) | No threshold; reported only |

Every threshold is a simple, deterministic `_min` (≥) / `_max` (≤) comparison. An **unknown threshold key** in a scenario is an immediate error (exit 2).

## Running

```bash
# One scenario + a given transcript
python benchmark/drift/run_drift.py \
  --scenario benchmark/drift/scenarios/long_session_basic.json \
  --transcript benchmark/drift/transcripts/good_session.jsonl

# Run every scenario under scenarios/ with its own default transcript
python benchmark/drift/run_drift.py --all

# Optional: write the summary to an explicit path (by default it only prints and writes no results directory)
python benchmark/drift/run_drift.py --all --json-out /tmp/drift_summary.json
```

Exit codes: `0` all thresholds met · `1` a threshold was not met · `2` missing or malformed input.

Apart from `good_session.jsonl` (a long session that meets every threshold), each `bad_*.jsonl` in `transcripts/` is a minimal replay that
**triggers exactly one kind of drift** (so tests can assert that the matching metric fails): plan changed without consent / invented questions / checkpoint reset / lost provenance labels / lost progress rows / wiki read outside the current chapter.

## Live-agent session log adapter

T5b adds a tiny stdlib-only adapter so live-agent pilots can be captured in a UTF-8 Markdown log first, then
converted to the JSONL shape above. This avoids hand-authoring JSONL and reduces Windows/PowerShell Unicode
pitfalls around Chinese text and emoji provenance labels.

```bash
# Inspect the starter format
python benchmark/drift/convert_session_log.py --template \
  benchmark/drift/templates/live_session_template.md

# Validate only
python benchmark/drift/convert_session_log.py \
  --in /tmp/live_session.md \
  --check

# Convert to explicit temp output, then score with T4
python benchmark/drift/convert_session_log.py \
  --in /tmp/live_session.md \
  --out /tmp/live_session.jsonl

python benchmark/drift/run_drift.py \
  --scenario benchmark/drift/scenarios/long_session_basic.json \
  --transcript /tmp/live_session.jsonl \
  --json-out /tmp/live_metrics.json
```

The adapter reads and writes UTF-8 explicitly, exits `2` on malformed input, and does not write tracked outputs
unless you explicitly point `--out` into the repository. It also validates supported event names (`read_file` /
`write_file`) and requires matching `files_after` snapshots when a turn records writes to `study_plan.md` or
`study_progress.md`. See [`docs/live_agent_pilot.md`](docs/live_agent_pilot.md) for the live pilot runbook and
commit boundaries.

## T5c · One-command real-agent smoke test (opt-in)

`run_live_smoke.py` automates the manual T5b pilot: **drive a real agent → record (in T5b format) → convert to JSONL → score with T4**, all in one command, and the exit code is the verdict (0 thresholds met / 1 drift detected / 2 gate or input error / 3 budget exceeded or failure abort).

```bash
RUN_SKILL_DRIFT_LLM=1 python benchmark/drift/run_live_smoke.py   --agent-cmd "claude -p {prompt}" --out-dir /tmp/live_smoke
```

- **Gate**: running any agent command requires `RUN_SKILL_DRIFT_LLM=1` (it incurs real call costs); CI never runs it.
- **Budget/abort**: `--max-turns/--max-output-chars/--max-prompt-chars/--turn-timeout`; if any limit is exceeded or the agent fails, it exits 3. An incomplete session is never scored as a clean one.
- **Honest division of labor**: the agent is invoked once per turn and can only "talk." File-writing behavior (progress persistence, writing plan changes to disk) is **not** covered by this smoke test (the deterministic replay layer guards it). What it tests is the **text-observable contract**: goal retention, quizzing from the bank ([#id] + no invented questions), provenance labels, and checkpoint language. Ten turns is a pilot, not statistical proof.
- **Scoring scenario**: `scenarios/live_smoke_basic.json` **contains only text-observable thresholds**. Wiki lazy loading, out-of-chapter reads, and progress-row persistence cannot be observed from one-shot text calls, so they **deliberately have no thresholds** (the deterministic replay layer covers them). This keeps a vacuous pass from being misread as verified.
- **The checkpoint is driven by the script (by design)**: the turn script plays the "student/environment." When the student says they are moving to Phase 2, the environment updates the progress file. The agent's compliance with the checkpoint is measured by the reset/goal metrics, and **state never advances based on the agent's verbal reply** (semantic interpretation is left to a future LLM judge).
- **Sandbox workspace**: each run copies the fixture to `<out-dir>/workspace` and makes it the agent's CWD, so reads and writes by a tool-using agent land in a throwaway copy and never touch the committed fixture (give the program in `--agent-cmd` as an absolute path or a command on PATH). The question-bank summary includes **options and reference answers** (so the scoring probes do not depend on the model's prior knowledge).
- Turn script: `templates/live_smoke_turns.json`; golden sample (produced by a local fake agent, self-authored): `fixtures/live_logs/live_smoke_golden.{md,jsonl}`. A clean checkout is enough to reproduce the "convert → score" half.

## Boundaries and limitations (stated honestly)

- **Deterministic replay ≠ real agent behavior**: the detectors hold only for scripted transcripts; whether a real model behaves this way is **not verified**.
- **Real-LLM long sessions have graduated to opt-in (B3)**: `run_drift.py --llm` is no longer a skeleton. It delegates to the real pipeline in `run_live_smoke.py`
  (drive a real agent turn by turn → record a T5b log → convert to JSONL → score with **the same** `compute_metrics`/thresholds → record in the ledger);
  token caps, failure aborts, and the fixture sandbox all live in the live runner. It requires `RUN_SKILL_DRIFT_LLM=1` (no env → exit 2;
  env set but no `--agent-cmd`/`--turns` → the delegated live runner exits nonzero because required arguments are missing; it **never reports success without an agent**).
  Usage: `RUN_SKILL_DRIFT_LLM=1 python run_drift.py --llm --agent-cmd "claude -p {prompt}" --out-dir <dir> --turns <spec>`.
- **The paid part stays out of CI**: this harness's deterministic layer costs nothing and runs in the root-level tests; the real-agent `--llm` path incurs real call costs
  (against a daily quota), so it is **triggered manually and never runs in CI**.

### The metrics are smoke heuristics, not semantic scorers: known inherent limitations

After an adversarial self-review, the items below are **inherent limitations of regex/structural heuristics**. Deterministic methods cannot fully fix them; semantic judgment is left to the future opt-in `--llm`:

- **Goal retention** is a **keyword blocklist**: it only catches off-topic phrasing listed in `unrelated_goal_phrases`; a different phrasing that is not on the list goes undetected.
  As a positive signal, the optional threshold `goal_marker_min` (requiring the assistant to mention the exam goal at least once) adds a floor, but it is still not semantic judgment.
- **Provenance labels**: `has_content_label` cannot tell "a one-line legend definition (🟢 来自资料:表示… ("🟢 From your materials: means…"))" apart from "actually labeling a specific sentence,"
  so a one-line legend counts as labeled (the same limitation as T2). Real sentence-by-sentence label checking is a job for an LLM.
- **Invention rate** depends on the skill's `[#id]` labeling convention: a question invented entirely in **prose** with no `[#id]` at all can only be caught
  by the structural signal "asked to quiz, but zero `[#id]`" (recorded as `untagged_questions`); it cannot decide sentence by sentence whether a piece of prose is an "invented question."
- **Checkpoint reset** only recognizes "阶段/phase + a number" or the restart words in `RESTART_PHRASES`; a purely natural-language 「从头再过一遍」 ("let's go through it all again from the start") that contains none of these cues is missed.
- **Plan authorization** relies on nearby keywords: `--` authorization only checks whether "the turn that made the change, or the user message right before it" contains a word like `改计划` ("change the plan"); it cannot understand
  **scope-limited** instructions such as 「只改错别字、别动顺序」 ("only fix typos, don't change the order") and will treat the adjacent change as authorized. What has been fixed is the **session-level** latch (one authorization letting everything through for the rest of the session).

**These are not bugs but the boundaries of deterministic replay.** On **labeled scripted transcripts** the harness makes structural assertions; adversarial evasion is left to
a future LLM judge. By contrast, the following **structural evasions are closed** (see the regressions in `tests/test_drift_harness.py`): out-of-phase quizzing and out-of-chapter reads
**no longer** stop being detected when `phase_context` is missing (they use **the current phase as it rolls forward within the session**, and phase↔chapter follows the **study_plan mapping**, not `chNN==阶段`);
checkpoint reset is checked on **every** resume turn and takes the **lowest** phase mentioned ("we're at Phase 2 now, but let's start from Phase 1 first" also counts as a reset); explanation-turn detection **cannot** be bypassed
through the `kind` value; progress rows are tracked by `[#id]` (rewording does not count as loss); plan authorization is not a session-level latch; a question that carries a real `[#id]` but **whose text has been swapped for an invented question** is judged invented by the same content comparison as T2;
extra unnumbered questions in mixed turns also count toward `untagged`; empty-run transcripts with only user turns and no assistant turns, as well as broken fixture JSON, both exit 2.
**Parsing is compatible with both self-authored fixtures and the real `scripts/ingest.py` template** (tables / checkboxes / checkpoint lines / mistake and confusion table rows).
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall