ZeKaiNie/universal-examprep-skill / full
universal-exam-cram-coach-full
Verified帮助学生在临考前进行结构化极速复习:解析课程资料/大纲/重点,按章节生成 wiki 知识库与标准题库,组织针对性刷题与判分,并记录复习进度和错题。当用户即将考试、需要快速复习计划、练习题、错题复盘或考前小抄时使用(关键词:期末/备考/复习/刷题/划重点/错题;exam, cram, study plan, quiz, review)。不适用于长期学习规划、与考试无关的写作或编程任务。
benchmark/drift/docs/live_agent_pilot.md
View on GitHub# Live-Agent Pilot Runbook
This runbook is for tiny, controlled live-agent calibration pilots that produce a replayable transcript for
`benchmark/drift/run_drift.py`. It does not replace the deterministic T4 replay tests, and it is not a full
benchmark.
## What A Live-Agent Pilot Means
A live-agent pilot is one short tutoring session run against a known, self-authored fixture. The goal is to
capture what the agent actually said and what files/tools it appeared to use, then convert that record into
T4 JSONL for deterministic scoring.
Keep it small:
- one session
- roughly 5-10 turns
- self-authored fixtures only
- no paid/full benchmark
- no external API calls unless a future task explicitly approves them
- stop immediately if the session is clearly broken
> **Automated mode (T5c)**: `run_live_smoke.py` runs this whole pilot as ONE command
> (drive → record → convert → score); this runbook stays the manual/fallback path. See README §T5c.
## Three Modes
| Mode | What Runs | What It Proves |
| :-- | :-- | :-- |
| Deterministic replay | Existing JSONL under `benchmark/drift/transcripts/` | The harness catches known drift patterns in scripted transcripts. |
| Codex self-run structured pilot | Codex writes a small session log from its own controlled tutor responses | The workflow can be manually exercised, but it is not an independent black-box model run. |
| True black-box agent pilot | A separate live agent is prompted and recorded | The external agent's observed behavior on that small session. Still not statistical proof. |
## Recommended Fixture
Start with:
```text
benchmark/drift/fixtures/mini_course_long
benchmark/drift/scenarios/long_session_basic.json
```
This fixture is self-authored and already aligned with the T4 drift scenario. Do not use private course
material, EEC160 material, or copyrighted lecture content for committed fixtures.
## Fill The Markdown Session Log
Use the template:
```bash
python benchmark/drift/convert_session_log.py --template \
benchmark/drift/templates/live_session_template.md > /tmp/live_session.md
```
On Windows, prefer opening `benchmark/drift/templates/live_session_template.md` in a UTF-8 capable editor and
saving a copy under `%TEMP%`, for example:
```powershell
Copy-Item benchmark\drift\templates\live_session_template.md $env:TEMP\live_session.md
```
Each turn uses this shape:
````markdown
## Turn 1
kind: resume
phase_context: 1
### User
我回来了,继续上次的复习。
### Assistant
🟢 来自资料:当前阶段是阶段1,继续栈/队列复习。
### Events
- read_file: references/wiki/ch1_stack_queue.md
- write_file: study_progress.md
### Files After: study_progress.md
```text
# 复习进度(study progress)
当前阶段:1
```
````
Supported turn fields:
- `kind` optional; common values are `resume`, `quiz`, `explanation`, `confusion`
- `phase_context` optional integer
- `tokens_in`, `tokens_out`, `cost_usd` optional non-negative accounting fields; `cost_usd` must be finite
Supported sections:
- `### User` required
- `### Assistant` required
- `### Events` optional, with lines like `- read_file: references/wiki/ch1_stack_queue.md`
- `### Files After: path` optional, with a fenced code block containing the full file snapshot
If user or assistant text needs to show this adapter syntax literally, put the example inside a fenced code
block so reserved headings such as `### Events` stay message content.
If a `Files After` snapshot itself contains a fenced code block, use a longer outer fence such as four
backticks so the inner triple-backtick block stays part of the snapshot.
Supported event types are deliberately narrow: `read_file` and `write_file`. Typos such as `readfile` are
malformed and the converter exits with code `2`.
Tracked writes must include snapshots in the same turn. If `### Events` records `- write_file:
study_progress.md`, `- write_file: study_plan.md`, or `- write_file: study_state.json`(A4 结构化
状态,`update_progress.py` 的官方写入), include the matching `### Files After: <该文件>` block.
Without that snapshot, T4 cannot score progress-row persistence, plan drift, or the A4 checkpoint
accurately, so the converter rejects the log. For these tracked files, `./` prefixes and
Windows-style backslashes are normalized to the canonical `study_progress.md` / `study_plan.md` /
`study_state.json` keys before JSONL is emitted.
## Convert To T4 JSONL
Validate only:
```bash
python benchmark/drift/convert_session_log.py \
--in /tmp/live_session.md \
--check
```
Convert:
```bash
python benchmark/drift/convert_session_log.py \
--in /tmp/live_session.md \
--out /tmp/live_session.jsonl
```
The converter reads and writes UTF-8 explicitly, preserves Chinese text and emoji provenance labels, and exits
with code `2` for malformed input.
## Run T4 Scoring
```bash
python benchmark/drift/run_drift.py \
--scenario benchmark/drift/scenarios/long_session_basic.json \
--transcript /tmp/live_session.jsonl \
--json-out /tmp/live_metrics.json
```
Do not lower drift thresholds just to make a tiny converter fixture pass. If a tiny log is only meant to test
the adapter, test loader compatibility and document that it is not a full drift pass.
## Windows And UTF-8
Avoid legacy shell encodings when the log contains Chinese or emoji labels such as `🟢 来自资料` and
`🟡 AI补充,可能与你老师讲的不完全一致`.
Safer habits:
- use explicit `--out` files instead of shell redirection for JSONL output
- use a UTF-8 capable editor
- avoid piping emoji/Chinese through legacy Windows PowerShell encodings
- if piping in PowerShell is unavoidable, set UTF-8 first:
```powershell
[Console]::OutputEncoding = [System.Text.UTF8Encoding]::new($false)
$OutputEncoding = [System.Text.UTF8Encoding]::new($false)
```
## Token And Spend Caps
Before running a true black-box pilot, decide the cap in writing:
- maximum one session
- maximum 10 turns unless a task explicitly says otherwise
- no model variants
- no full benchmark matrix
- no private materials
- no API keys in logs, prompts, or transcripts
If the environment cannot run a separate black-box agent, say so and record the run as a Codex self-run
structured pilot rather than pretending it was independent.
## What Not To Commit
Do not commit:
- private live-run transcripts
- copyrighted course material
- EEC160 materials
- raw external model outputs from private sessions
- API keys, account identifiers, or secrets
- generated metrics from ad hoc live pilots unless explicitly approved
Small self-authored fixture logs under `benchmark/drift/fixtures/live_logs/` are okay when they are designed
for tests and contain no private material.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |