DAAF-Contribution-Community/daaf / .claude/skills/daaf-deploy-smoke-testing
daaf-deploy-smoke-testing
UnverifiedDAAF deployment smoke test for maintainers: verifies that a LIVE install works end-to-end in one of five install routes (anthropic-subscription, anthropic-api, chatgpt-subscription shim, openai-api shim, openrouter), named with the required --route flag. Tiered probes: Tier 0 free preflight (image flavor, route, env and credential coherence, versions, hooks, settings invariants, shim health, framework integrity); Tier N free network checks; Tier D free deterministic battery (bats, Pester, lint, safety hooks, shim unit suite, browser services); a preflight gate, then paid Tier 1 round-trip, Tier 2 battery (dispatch, Python and R coding, web, skills, guardrails), Tier A Auto mode and Tier 3 golden path. Every run builds a scrubbed package to carry to the main container, where --import writes a cross-route MATRIX.md for triage. Use before a release, after a rebuild, update or provider change, to check Auto mode, or to triage a smoke package. Runs on any DAAF image. NOT DAAFBench and NOT scripts/smoke_tests/.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 28.8 KB |
| references/interpreting-results.md | markdown | 126.9 KB |
references/interpreting-results.md
View on GitHub# Interpreting Deployment Smoke Results
Probe-by-probe interpretation for the DAAF deployment smoke suite: what each verdict means, which signals are expected mechanics rather than defects, route-conditional expectations, and how to route a real failure to its documented fix. Read this alongside a `report.md` from `scripts/deploy_smoke/reports/{local timestamp}_{stated route}/`, or from an imported package under `scripts/deploy_smoke/inbox/<package>/`. Each probe row below is what the report's problems table points to when a probe has no more specific fix location. Thresholds and match strings that no live run has confirmed are listed at the end.
## Verdict Semantics
| Verdict | Meaning | Affects exit code? |
|---------|---------|--------------------|
| **PASS** | The probe's machinery worked as expected. | no |
| **FAIL** | A real defect for this route. Fix before trusting the deployment. | **yes** (nonzero) |
| **WARN** | Something worth a look, but not necessarily broken. Often expected on short headless runs (see below). | no |
| **SKIP** | Not applicable to the detected route (e.g. shim `/health` on a non-shim route). Correct behavior. | no |
| **INFO** | Informational context (e.g. the model-family/ceiling-hook posture), not a pass/fail signal. | no |
Only **FAIL** flips the overall verdict, matching the `run_all_smoke_tests.sh` contract. When triaging a report, read every FAIL first, then scan WARNs against the "expected mechanics" notes below before treating one as a defect.
The suite is **capability-structural**: Tier 1–2 probes check whether the machinery worked (did a subagent dispatch, did a script run, did a Skill load), *not* whether the model followed DAAF's protocols with high fidelity. Adherence quality is DAAFBench's job. A probe that PASSes confirms the plumbing is intact, not that the model behaves ideally. Tiers A and 3 come closest to behavior: Tier A checks that Auto mode runs a command, and Tier 3 checks that the orchestrator engages on a real question and that the golden path works end to end (a resumed session answers, with a curated skill retrieved). Tier 3 records the reply's mode-gate behaviour without grading it. Read their transcripts before treating a FAIL or WARN there as a deployment defect.
**Synthetic records.** Six probe IDs are written by the runner rather than a probe: `T0.ABORT` (FAIL, name "Paid tiers skipped (preflight gate)": a preflight gate probe FAILed, so no paid tier ran; the detail names the blocking probe IDs and the evidence lists every gate verdict, see § The Preflight Gate. Named "Paid tiers skipped (profile gate)" and carrying a profile, it means one `--profiles` overlay failed its profile gate and only that profile's paid tiers were skipped), `T<tier>.CRASH` (FAIL: that tier raised an exception; the evidence holds the last 40 traceback lines, and the fix is in `scripts/deploy_smoke/`, not the install), `POST.CRASH` (FAIL: the post-run shim-log scan raised an exception, same reading), `RUN.CRASH` (FAIL: the whole tier block stopped early, for one of two reasons. An **interrupt**: Ctrl+C (SIGINT, exit 130), SIGTERM from `timeout`, `kill` or a CI cancellation (exit 143), or SIGHUP from a closed terminal (exit 129); after SIGTERM or SIGHUP the detail reads `received SIGTERM` or `received SIGHUP`, and the hint says so. An interrupt is not a defect: re-run the tiers that did not finish. Or an **unhandled exception** in the runner's own tier-block code, which no tier's guard wraps (for example the per-profile T0.14 model re-check or the preflight-gate bookkeeping; an exception inside a tier is that tier's `T<tier>.CRASH`): the detail names the exception, the exit code is 1, and, as for `T<tier>.CRASH`, the fix is in `scripts/deploy_smoke/`. Running probes cleaned up first, TD.9 stopping its services, and the report and package still cover the probes that ran. After SIGHUP, closing console lines bound for a terminal are discarded even when it is still open, as after a `kill -HUP`, so read the report folder. Handling starts after the cost-confirmation prompt: an interruption at the prompt, or while versions are collected just after it, leaves no report, no package and no `RUN.CRASH`, because no probe has run), `T<tier>.UNAVAILABLE` (SKIP: the checkout has no entry point for Tier N, A or 3) and `POST.UNAVAILABLE` (SKIP: the checkout has no `probe_shim_log_scan`).
## Reading a Report: the Diagnosis Summary
`report.md` opens with a Diagnosis summary written for the main session that will fix things:
1. **Identity block.** Stated and detected route (a `MISMATCH` here is the T0.1 FAIL), DAAF branch, `git describe --dirty`, SHA and dirty-file count, Claude Code version, shim version (shim routes, from T0.8's `/health`), Codex CLI version, the model mapping as configured, tiers and profiles, the **Preflight gate** row (`not applied (no paid tier selected).`, `passed: …` with the probes reused from Tier 0 or run as the gate subset, `**BLOCKED** by …`, or `**FAILED** (…); paid tiers ran anyway…` when `--allow-preflight-fail` overrode it; with named `--profiles` it adds `Per profile (T0.1, T0.3, T0.16, T0.17 re-checked on each overlay): …`, one `passed`, `**BLOCKED** by …` or `**FAILED** (…), ran anyway` per profile), start and end time (UTC), paid calls and cost, and the overall verdict with counts. The versions come from `smoke_probes.collect_versions`, the same source as T0.15, so the dirty count matches T0.15's: it covers tracked **and untracked** files (`git status --porcelain`; `versions.daaf_dirty_files` and `versions.daaf_dirty_scope` in `report.json`). Paid calls are counted per `claude -p` invocation, so T3.1 and the T3.2 call that resumes its session count as two. Cost is the sum of what those invoking probes report, or T1.7's `total_cost_usd` when none does; "not reported" means no probe captured a cost. When a run used `--resume` (Tier 3), the total is labelled an **upper bound** (`total_cost_is_upper_bound: true` in `report.json`): a resumed session's reported cost may include the earlier call's cost, so the sum could count T3.1 twice. That cumulative-cost behaviour is inferred, not observed.
2. **Problems table**, every FAIL then every WARN, one row each. Columns: *Probe* (ID, profile, name); *Verdict*; *Observed* (the probe's detail, cut to 200 characters; when a live-run warning turned a PASS into WARN, the detail opens with it, as the live-run health gate under Tier 1 describes); *Likely cause* (the probe's hint, a static text per probe in `smoke_probes.PROBE_HINTS` / `route_detection.ROUTE_PROBE_HINTS`, or a shape-specific hint such as T0.8's stale-daemon text); *Evidence in package* (`report.md § <probe>`, the probe session's `transcripts/<sid8>/` folder, the packaged log for log-based probes, and any report-relative evidence file the probe names); *Where to fix* (a file, doc or action from the runner's location map, or this file's row for the probe; every `*.CRASH` record points at `scripts/deploy_smoke/` and the synthetic-records paragraph of § Verdict Semantics). The log-based probes point at `logs/shim.log` (T0.8, T0.18, T0.20, TA.1) and `logs/codex_config.log` (T0.18). In the **packaged** `report.md`, an evidence path the zip does not hold is marked `(withheld)` (the post-scrub or names gate removed it) or `(not in package)` (never staged: not found, excluded, off-route or past the size cap).
3. **Passed** line: PASS count out of the probes run, per tier (plus a **Post-run** entry when the post-run scan ran), with SKIP and INFO counts.
The fingerprint table and full per-probe detail follow, each live probe with its duration, session ID prefix and cost, and a **Hint** line on FAIL and WARN. The report writer strips raw ANSI escape sequences and control characters from every name, detail, hint and evidence block; the statusline probes (T0.7, T1.6) also strip the escaped text forms before quoting (see T0.7). `report.json` (schema version 2) carries the same data plus `started_utc`, `ended_utc`, `stated_route`, `versions{}`, `paid_calls`, `total_cost_usd`, `total_cost_is_upper_bound`, `allow_preflight_fail` (bool), `preflight_gate{}` (`applied`, `allow_preflight_fail`, `probes`, `reused_from_tier0`, `gate_subset_run`, `blocking`, `overridden`, `paid_tiers_skipped`, `per_profile`; only the first two when no paid tier was selected. `per_profile` maps each named `--profiles` profile to `probes`, `blocking`, `overridden`, `t014_verdict` and `paid_tiers_skipped`, true when its profile gate blocked it or its T0.14 re-check FAILed; it is `{}` without named profiles or when the ambient gate blocked), and per probe `tier` (`post` for the post-run scan), `duration_s`, `session_ids`, `cost_usd` and `hint`.
---
## Tier 0 — Free Preflight (no LLM)
| Probe | PASS means | FAIL means / how to fix |
|-------|-----------|-------------------------|
| **T0.0 Image flavor** | **INFO**, never FAIL and never a gate. The flavor is read from the tools on `PATH`, not from a variable: `developer image` when `bats`, `pwsh`, `shellcheck` and `gh` are all present, `standard image` when none is, `partial` otherwise (the detail names the missing ones). The suite runs on every flavor; on the standard image Tier D SKIPs TD.1 (bats) and TD.2 (Pester). The `DAAF_DEV` value is recorded as context only. The same flavor is handed to T0.15 and T0.22. | **WARN** = `DAAF_DEV=1` is set but a developer tool is missing: the variable was set without a rebuild. The detail names only the Tier D batteries that will SKIP as a result: TD.1 when `bats` is missing, TD.2 when `pwsh` is. A missing `shellcheck` or `gh` SKIPs no battery, and the detail says what it changes instead: without `shellcheck`, TD.1 still runs but skips its shellcheck lint tests (a TD.1 PASS then does not cover them); no Tier D battery uses `gh`. Rebuild (`bash rebuild_daaf.sh` from `daaf-docker`) so the build argument takes effect, or remove `DAAF_DEV=1` from `environment_settings.txt` if you meant the standard image. |
| **T0.1 Route detection + assertion** | Detected route reported; matches `--route` if asserted. | `--route` asserted a route the live env does not produce. Detection is authoritative — the expectation is wrong or the env is misconfigured. Reconcile the env vars in the fingerprint against the intended route. |
| **T0.2 Model family** | INFO — reports family (claude/gpt/glm/unknown; tier aliases in `ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_MODEL` and `CLAUDE_CODE_SUBAGENT_MODEL` resolved through their mappings) and the expected ceiling-hook posture. The `enforce-model-ceiling.sh` that ships with this release stands down only for an enabled `CLAUDE_CODE_SUBAGENT_MODEL_FORCE` (`1` or `true`, rule a) or an OPUS/SONNET remap (rule b); `CLAUDE_CODE_SUBAGENT_MODEL` alone does not stand it down. | Never FAILs. Use it to confirm the ceiling-hook expectation matches intent (see route sections below). T0.2 reads the installed hook (read-only) and quotes whether it carries the force flag. If it does not, the install has an older hook that stands down whenever `CLAUDE_CODE_SUBAGENT_MODEL` is set and does not recognize `CLAUDE_CODE_SUBAGENT_MODEL_FORCE`; the detail then says so and describes that hook's posture. If the hook cannot be read, the current rule set is assumed and the evidence says so (T0.6 covers registration). |
| **T0.3 Env coherence** | Required vars for the detected route are present and mutually coherent. | Route-specific missing/incoherent var. The `detail` names each problem; see the route sections below for the exact fix. |
| **T0.4 Context-window declaration** | The window resolves natively, or the applicable route-specific check passes. On the direct OpenAI API shim and OpenRouter routes, any valid canonical positive explicit `CLAUDE_CODE_MAX_CONTEXT_TOKENS` value satisfies T0.4; exact equality to `1050000` is not enforced. With no explicit declaration, a supported `[1m]` hint is sufficient only on the direct OpenAI API shim. OpenRouter requires a canonical explicit declaration when its wide window is not otherwise resolved natively, which T0.4 judges selector by selector: any non-Claude selector (tier aliases in `ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_MODEL` and `CLAUDE_CODE_SUBAGENT_MODEL` resolved through their mappings, matched case-insensitively; `inherit` and `default` name no model and are skipped, `opusplan` is checked through both the OPUS and SONNET mappings, and `best` through the FABLE one (an inference: Claude Code's docs define `best` only for Claude apps gateway sessions); an `ANTHROPIC_DEFAULT_MODEL` value that Claude Code ignores, see T0.14, is not inspected) requires it, even when another selector names a Claude model; the recommended GPT configuration is a provider-prefixed bare slug such as `openai/gpt-5.6-sol` or `openai/gpt-6-astra` plus `CLAUDE_CODE_MAX_CONTEXT_TOKENS=1050000`, but T0.4 does not enforce that slug form when a valid explicit declaration is present. On the exact ChatGPT-subscription (Codex) shim lane, mapped GPT flagship configurations require a canonical positive explicit declaration no greater than `919000`. | A required declaration is absent, or an explicit declaration is invalid or non-canonical. This includes an unresolved OpenRouter wide-window model with no canonical explicit declaration (`[1m]` alone is insufficient), and an exact ChatGPT-subscription mapped GPT flagship with a missing, invalid, non-canonical, or greater-than-`919000` declaration. On the direct OpenAI API shim, an unresolved model fails only when it has neither a valid canonical explicit declaration nor a supported `[1m]` hint. Claude Code otherwise silently assumes ~200k and under-reports a wide API/OpenRouter window; `[1m]` cannot raise the measured ChatGPT-subscription backend ceiling of 919,000 tokens, measured 2026-09-05 for both `gpt-5.6-sol` and `gpt-6-astra`, so use `CLAUDE_CODE_MAX_CONTEXT_TOKENS=919000` on the ChatGPT-subscription lane for either model. A user policy decision of 2026-09-29 (not validation) also maps `gpt-6-sol`, `gpt-6-luna`, and the flagship `-pro` forms (for example `gpt-6-sol-pro`, `gpt-6-astra-pro`, `gpt-5.6-sol-pro`) as GPT flagships, so T0.4 applies the same `919000` requirement to them. A further user policy decision of 2026-10-12 adds `gpt-6.1-sol` (and its `-pro` form) on the same terms. `gpt-6.1-sol`'s cap was measured on 2026-10-13: it accepted 919,744 tokens and rejected a request of about 921,973 tokens. For the other models the cap is inferred from the Sol/Astra measurements. `gpt-6-terra`, bare `gpt-6`, and GPT-6 mini/chat stay unmapped. Recommended authoritative values remain `1050000` for the supported 1M GPT API/OpenRouter configuration (including `gpt-6-astra`) and `1048576` for exact `z-ai/glm-5.2`. Physical capacity does not change DAAF's model-family quality thresholds. |
| **T0.5 CLI available** | `claude --version` responds. | CLI missing/broken in the container. Rebuild; verify the image. |
| **T0.6 Hook registration** | Every `.claude/hooks/*.sh` is registered in `settings.json` (or is a known per-agent-frontmatter hook: `enforce-file-first.sh`, `adjudicator-read-scope.sh`). One file is exempt by exact name: `bash-safety.site.sh`, the site file that `bash-safety.sh` sources and that must not be registered; the evidence lists it as "sourced by a hook" (`smoke_probes._SOURCED_HOOK_FILES`; any other `*.site.sh` is checked like a hook). | A hook script is present but unregistered. A shell overwrite of `settings.json`, or a partial deploy, dropped a registration. Restore the hook chain in `settings.json`. |
| **T0.7 Statusline rendering** | Both `context-bar.sh` and `subagent-bar.sh` render non-empty output against a synthetic payload (session id `smoke-synthetic`). Both render with `DAAF_CONTEXT_BAR_CACHE_DIR` and `DAAF_SUBAGENT_BAR_CACHE_DIR` pointed at the report's `evidence/t07_statusline_cache/`, so their session caches land report-local, never in `/tmp`. The evidence lists the caches written there, and quotes a read-only before/after `stat` of the exact `/tmp/claude-*-smoke-synthetic` paths. The rendered output is quoted after `route_detection.strip_ansi`, which removes ANSI colour codes both as raw ESC bytes and as escaped text (`\u001b[…`, `\x1b[`, `\033[`, `\e[`): `subagent-bar.sh` emits JSON, so its colour codes arrive as the six characters `\u001b`. | FAIL = a statusline crashes or emits nothing. These are fail-open scripts (a crash still exits 0), so empty output is the real signal — inspect the script against the synthetic payload schema. WARN = both render, but a synthetic-session `/tmp` cache was created or modified during the render: a renderer does not honour its cache-dir override. Check the `*_CACHE_DIR` default line in `context-bar.sh` / `subagent-bar.sh`. |
| **T0.8 Shim /health** | (shim routes) `/health` answers with `service=daaf-anthropic-openai-shim`, `status=ok`, the route's `backend_mode`, a well-formed `version` **equal to `SHIM_VERSION` in the shim source on disk** (the same rule `start_shim.sh` uses), `codex_home_present=true` on the ChatGPT lane, and a `gpt_service_tier` block with the exact key set (`backend_mode`, `requested_tier_vocabulary` = `priority`, `native_fast_disabled` = `true`, a coherent `policy`, a bounded `latest_terminal`). The evidence keeps an allowlisted projection only, and the detail says the requested Fast/Priority policy is not proof of the served tier. SKIP on other routes. | **Stale daemon** (the detail leads with "running shim is stale"): the daemon still serves older code after an update, so every shim fix looks absent (F6). Restart it yourself: `bash /daaf/scripts/provider_shim/start_shim.sh --restart`, then re-run Tier 0. Auto mode may refuse an agent-initiated restart. Unreachable → the shim is not running (`start_shim.sh`). `backend_mode` mismatch → wrong `SHIM_BACKEND_MODE`. `codex_home_present=false` → see T0.9. A `gpt_service_tier` problem → a shim older than v1.3.9 or a changed `/health` schema; the check fails closed on any missing or extra key, so a deliberate schema change must update `smoke_probes._validate_gpt_service_tier_health` too (`FRAMEWORK_INTEGRATION_CHECKLIST.md` PM2). |
| **T0.9 ChatGPT auth validity (/health)** | (ChatGPT lane) the shim's `/health` `auth` block reports `state=valid`; the shim derives it from `auth.json` presence and the access token's expiry and never exposes token material. Supplementary evidence records whether `$CODEX_HOME/auth.json` is readable (contents never read). SKIP otherwise. | **WARN** = `state=expiring` (the detail gives `days_left`): run `codex login --device-auth` before it lapses. **FAIL** = `expired`, `absent` or `unreadable` (re-run `codex login --device-auth`, then restart the shim), `CODEX_HOME` unset, `/health` unreachable, no `auth` block (a pre-v1.3.0 shim), or an unexpected state such as `n/a` on this lane. |
| **T0.10 Workspace invariants** | `check_workspace_invariants.sh -q` clean (no unauthorized symlinks / repo-root leak artifacts / names Windows cannot create or case-only name clashes). | An invariant violation exists on the live filesystem. Run `bash /daaf/scripts/check_workspace_invariants.sh` (verbose) to see the offending path. |
| **T0.11 R UTF-8 locale** | `Rscript -e 'quit(status = !isTRUE(l10n_info()[["UTF-8"]]))'` exits 0 — R starts under a UTF-8 locale (codeset UTF-8, MBCS on). Confirms the image's `LANG`/`LC_ALL=C.UTF-8` is in effect, so R reads multibyte UTF-8 (e.g. `yaml::read_yaml()` on `mirrors.yaml`) correctly rather than silently returning NULL. Route-independent — an image property, not a provider one. Note: the harness strips the `LC_CTYPE` its own Python interpreter injects via PEP 538 coercion before spawning `Rscript` (when `LANG`/`LC_ALL` are unset), so the verdict reflects the true image env — an unsanitized child would false-PASS on a stale image. | FAIL = R's codeset is `ANSI_X3.4-1968`/`POSIX` — `LANG`/`LC_ALL` are unset in the running container (a stale or pre-v3.0.0 image), and R will silently corrupt UTF-8. Fix at the source: confirm the `ENV LANG=C.UTF-8` / `ENV LC_ALL=C.UTF-8` block near the top of the `Dockerfile` and rebuild (`bash rebuild_daaf.sh` from `daaf-docker`). Unlike Python, R has no PEP-538 startup coercion, so the image env is the only complete fix; the education-data-query skill's runtime `Sys.setlocale()` guard mitigates file reads but cannot repair parse-time literal mangling. SKIP = `Rscript` not on PATH; a hung `Rscript` (60s timeout) is a FAIL. |
| **T0.12 Local settings shadowing** | No `.claude/settings.local.json`, or one whose `env` block shares no key with `settings.json`'s `env`. Claude Code applies the local file over the committed one, and an `env` block follows that precedence. Local-only keys are listed in the evidence (they override the container environment, not `settings.json`). | **FAIL** = the local file sets a model selector (`ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_MODEL`, `ANTHROPIC_DEFAULT_OPUS_MODEL`, `ANTHROPIC_DEFAULT_SONNET_MODEL`, `ANTHROPIC_DEFAULT_HAIKU_MODEL`, `ANTHROPIC_DEFAULT_FABLE_MODEL`, `CLAUDE_CODE_SUBAGENT_MODEL`) to a different value than `settings.json`. That means chiefly `ANTHROPIC_DEFAULT_MODEL`, the only selector the shipped `settings.json` sets: every session then starts from the local default instead. Remove the key from the local file, or change it on purpose, and start a new session. **WARN** = another shadowed key, or a model selector pinned to the same value (harmless while the two match, but it drifts at the next model bump). Also WARN when the local file sets a model selector that `settings.json` does not set at all (e.g. `ANTHROPIC_MODEL`, a tier alias, or `CLAUDE_CODE_SUBAGENT_MODEL`): nothing is shadowed, but the local value still overrides the container environment for every session. (Since Claude Code 2.1.251 `CLAUDE_CODE_SUBAGENT_MODEL` reaches only subagents without their own model unless `CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1`.) Also WARN when the local file is invalid JSON or not an object, since Claude Code may ignore it. The evidence names the file and the keys; values are shown only for model selectors, through the suite's redaction helper. |
| **T0.13 claude.ai synced surface** | Always **INFO**; never FAILs. Lists the non-empty `~/.claude/skills/synced/*` and `~/.claude/plugins/synced/*` entries, resolved via `CLAUDE_CONFIG_DIR` or `$HOME/.claude`. It also names every hook event a synced plugin registers (from `hooks/hooks.json` or `.claude-plugin/plugin.json`), in the form "Plugin `<name>` registers PostToolUse, Stop hook(s)". Each plugin on disk is labelled `disabled`, `enabled` or `enabled state not set`. The label comes from the `enabledPlugins` object in `~/.claude/settings.json`, then `.claude/settings.json`, then `.claude/settings.local.json`, with later files winning per key. Disabling a synced plugin with `/plugin` was observed to write `"<name>@synced": false` to user settings. For a disabled plugin, the hook sentence says its hooks should not load. The per-key precedence across scopes is inferred. An error while listing these directories (e.g. a permission error) is reported in the detail, still as INFO. | Visibility only. DAAF **coexists** with claude.ai sync because a repository cannot turn it off. Synced plugins can register hooks, agents, and MCP servers; plugin hooks run outside DAAF's audited hook set, which is why they are named here. Finding files on disk doesn't prove a session loads them. To opt out, set `syncClaudeAiSkills` / `syncClaudeAiPlugins` to `false` in `~/.claude/settings.json` (which also moves existing copies to `.trash`) or in `.claude/settings.local.json`. |
| **T0.14 Main-model/route coherence** | Configuration-compatible under bounded inspected inputs, not provider acceptance. Invocation selector precedes effective `ANTHROPIC_MODEL`, then merged saved top-level `model`, then effective `ANTHROPIC_DEFAULT_MODEL` (Claude Code 2.1.236+; ignored when set to `default`, `inherit`, `opusplan` or `haiku`, in any letter case; reported with source `<layer>_settings_env_default_model` or `environment_default_model`). Short opus/sonnet/haiku/fable aliases (also in the `<alias>[1m]` form that `/model` saves, e.g. `opus[1m]`; matched case-insensitively, as in T0.4) expand only their own `ANTHROPIC_DEFAULT_<ALIAS>_MODEL` mapping; an unmapped alias is Claude Code's built-in Claude model for that tier (family `claude`, exact ID not resolved). The shipped `ANTHROPIC_DEFAULT_MODEL=opus` therefore PASSes on the native route. | FAIL = known incompatible main family/route (notably explicit Claude on a GPT shim, or an unmapped alias such as the shipped `opus` default on a shim route with no `ANTHROPIC_DEFAULT_OPUS_MODEL`), even if T0.2's combined family says GPT. WARN = unknown selector/dynamic default, malformed/unreadable/oversized settings, or managed-policy uncertainty. User/project/local settings env is projected in the observed 2.1.281 order, respecting `CLAUDE_CONFIG_DIR`; no auth-store reads or automatic configuration changes. It is not full client emulation. |
| **T0.15 Versions and identity** | `claude --version` equals the Dockerfile pin (`ARG CLAUDE_CODE_VERSION`) when that pin is readable, `codex --version` equals `ARG CODEX_VERSION` when that pin is readable, the git SHA resolves, and the dev binaries `bats`, `pwsh` and `shellcheck` are present. Branch, `describe --dirty` and the dirty-file count (tracked and untracked files) are recorded for the identity block but not judged: of the git identity, only a missing SHA WARNs. The pin parser takes the first matching `ARG` line and tolerates leading whitespace, a quoted value (`ARG X="1.2.3"`) and a trailing `# comment`; an unreadable pin skips its comparison. The evidence also records the shim source version. | **FAIL** = on a shim route, Claude Code is below the 2.1.276 floor (2.1.265-2.1.275 broke every shim turn; F31). **WARN** = the Claude Code CLI differs from its pin (judged only when the pin is readable; a stale or mixed install); the Codex CLI is missing or differs from `ARG CODEX_VERSION` (judged only when that pin is readable); `claude --version` is unavailable; `git rev-parse HEAD` gave no SHA; or a dev binary is missing on any image but the standard one (in practice a `partial` flavor, T0.0). On the `standard image` a missing dev binary is reported as INFO instead, since that image never carries them. Rebuild to the pins; commit or stash dirty files (untracked ones count) before a release run so the package's SHA means something. |
| **T0.16 Credential overlap** | (native Anthropic routes) exactly one credential for the stated route, judged by presence only (`unset`/`empty`/`set`, values never read). SKIP on OpenRouter and the shim routes, whose key rules are T0.3. | **FAIL** = `ANTHROPIC_AUTH_TOKEN` is set (it outranks the API key and the subscription login), or the stated route is `anthropic-subscription` while `ANTHROPIC_API_KEY` is set: in `-p` mode the key is always used, so every paid probe would test and bill the key. **WARN** = `anthropic-api` with `CLAUDE_CODE_OAUTH_TOKEN` also set (the key wins; the token is unused overlap). Keep one credential per install. |
| **T0.17 Tier-alias completeness** | Shim routes: all four `ANTHROPIC_DEFAULT_{FABLE,OPUS,SONNET,HAIKU}_MODEL` are set and none names a Claude model. OpenRouter with a non-Claude family: all four are set (Claude slugs are accepted, since OpenRouter serves them). SKIP on the Anthropic routes and on a Claude-family OpenRouter install. The values are judged as a session would see them: the process environment, overridden by the `env` blocks of the user (`$CLAUDE_CONFIG_DIR` or `~/.claude`), project and local `settings.json` files, later files winning (the observed 2.1.281 order, the same one T0.14 uses). Those reads are bounded and never touch an auth store; a missing or malformed file contributes nothing. The evidence names each value's source (`process env`, `user settings`, `project settings`, `local settings` or `unset`). | **FAIL** (shim routes) = an alias is unset or Claude-named, so Claude Code sends `claude-*` requests the shim cannot serve (the ChatGPT lane fast-fails them; F2, F3). Unmapped HAIKU is the usual culprit: it carries session naming and resume summaries. **WARN** (OpenRouter) = an alias is unset, so Claude Code asks for its built-in Claude name for that tier. Map all four in `environment_settings.txt` (see `environment_settings_example.txt`), recreate the container, start a new session. |
| **T0.18 Codex config state** | (shim routes) the last run recorded in `scripts/provider_shim/logs/codex_config.log` succeeded (summary exit 0, no refused or error lines). INFO when there is no log and no `DAAF_CODEX_CONFIG_*` line. The log holds key names and reasons, never values. | **FAIL** = the last render reported `ERROR:`, an error line, a refused entry (a name or value that looked like a credential) or a nonzero exit: fix the named `DAAF_CODEX_CONFIG_<N>` line, then `start_shim.sh --restart`. **WARN** = `DAAF_CODEX_CONFIG_*` is set but the log is absent (the renderer never recorded a run; F7), or the log has no summary line. |
| **T0.19 Prompt-cache health** | (shim routes) INFO with the cached/input token ratio from `/health.prompt_cache`, and "too few to judge" under 20 requests with usage. | **WARN** = at least 20 requests and a ratio below 0.3 (quota drains fast; F29; both numbers are unconfirmed, see the end of this file), or no well-formed `prompt_cache` block (a shim older than 1.3.6), or `/health` unreachable (see T0.8). Confirm the shim is 1.4.6 or later and watch the ratio over a longer session. |
| **T0.20 Shim-log scan (pre-run history)** | (shim routes) Always **INFO** in Tier 0. Counts only, never line text, over `scripts/provider_shim/logs/shim.log` since the current daemon start (counters reset at every `shim vX starting` line), which is history from before this run. The detail gives the counts and what the verdict would be ("Would read FAIL: …"), so an old signal is visible without failing a healthy install. An unreadable log is INFO too. SKIP on other routes. | Never FAILs or WARNs in Tier 0. The verdict comes from the **post-run** T0.20 scan of this run's window (see Post-run below), which uses the same signals. A "would read" FAIL or WARN here with a clean post-run scan means the signal predates the run: an old `x-codex-turn-state` line, for example, stays in the file until the log is rotated or trimmed (shim 1.4.9 stops writing them; F28). |
| **T0.21 Runtime identity and filesystem** | Non-root `appuser`, `/daaf` and `/daaf/research` writable, a probe file created and removed inside the report's `evidence/tier_0/`, `git status` works, at least 5 GB free. | **FAIL** = running as root, a folder not writable, the probe file failed, or `git status` failed (often `safe.directory`). **WARN** = under 5 GB free, or a user other than `appuser`. |
| **T0.22 Toolchain** | `python3`, `Rscript`, `quarto`, `jq`, `git`, `curl` and `uv` are present (first version line recorded), plus the dev tools `bats`, `pwsh` and `shellcheck`. | **FAIL** = a required tool is missing: rebuild the image. **WARN** = a dev-only tool is missing on any image but the standard one (in practice a `partial` flavor, T0.0; Tier D will SKIP it); it needs `DAAF_DEV=1` at build time. On the `standard image` a missing dev-only tool is **INFO**: that image never carries them. |
| **T0.23 Framework integrity** | `.claude/settings.json` parses and every registered hook command exists and is executable; every git-tracked `*.sh` with mode `100755` is executable on disk; every `.claude/skills/*/SKILL.md` has a valid `name` and a `description`; every agent's frontmatter parses and its `tools:` never includes `Agent` or `Task`; `CLAUDE.md` exists. | **FAIL** names the file: restore an executable bit with `chmod +x` (git's index mode alone does not fix the disk), repair frontmatter, or remove `Agent`/`Task` from an agent's tools. Hook and settings repairs are user-applied. |
| **T0.24 Settings invariants** | The settings DAAF relies on hold, read through the same layered reader as T0.14/T0.17 (process env, then the user, project and local `settings.json` `env`, later winning; the evidence names each value's source layer): `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH` is `1`; `CLAUDE_CODE_DISABLE_BACKGROUND_TASKS` is `1`; `CLAUDE_CODE_TMPDIR` is set in the **process** environment, absolute, not under `/tmp` (literal or resolved) and writable (a folder not yet created passes when its nearest existing parent is writable, since Claude Code creates it on first use); `statusLine` and `subagentStatusLine` are registered as `command` statuslines whose scripts exist and are executable. Claude Code 2.1.251 and later ignore `CLAUDE_CODE_TMPDIR` in a settings file, so a settings value is reported, never judged. | **FAIL** = spawn depth not `1` (the primary nested-dispatch layer; `block-nested-dispatch.sh` is only the backstop); `CLAUDE_CODE_TMPDIR` unset, relative, under `/tmp` (scratchpad files there are lost silently) or not writable; a statusline unregistered, not of type `command`, or its script missing or not executable. **WARN** = background tasks not disabled (the wave barrier stays binding either way, but inline subagent returns are the shipped assumption), or a settings layer present but malformed, unreadable or oversized, so its values were not read. Restore the shipped values: both flags in `.claude/settings.json` `env` (user-applied; look for an override in `settings.local.json` or `~/.claude/settings.json`, which the evidence names), `CLAUDE_CODE_TMPDIR=/daaf/claude-code-tmp` in `docker-compose.yml` (recreate the container), and the statusline commands pointing at the executable `.claude/scripts/context-bar.sh` / `subagent-bar.sh`. |
> **Tier 0 limitation — ambient env only.** Tier 0 runs **once**, against the ambient (as-launched) environment, before any profile overlay is applied. Its route/env-coherence and context-window checks (T0.1–T0.4) therefore describe the base installation, **not** any `--profiles` overlay. A profile that, say, sets `ANTHROPIC_MODEL` to a GLM slug and declares its window via `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is exercised only in that profile's live-tier runs: Tier 0 itself never re-validates per-profile overlays. Separately, before each profile's live tiers (1, 2, A and 3) the runner re-checks two things on that profile's actual child environment, ambient plus overlay (including when Tier 0 is omitted). **Main-model coherence** (T0.14) is re-checked for every profile. For a **named** `--profiles` profile, the **profile gate** also re-runs T0.1 (route detection), T0.3 (env coherence), T0.16 (credential overlap) and T0.17 (tier aliases), with the route information rebuilt from that environment, so T0.1 FAILs when the overlay moves detection away from the stated `--route` (for example by setting `ANTHROPIC_BASE_URL`) and T0.17 sees the profile's own model family; their names end in ` (profile gate)`. T0.4 and the other Tier 0 probes are not re-run per profile. A known incompatibility (T0.14 FAIL) or a profile-gate FAIL blocks that profile, not its compatible siblings; a profile-gate FAIL also records a `T0.ABORT` carrying the profile. WARN preserves ordinary exit behavior. The re-checks are filed under the first live tier selected (Tier 1 in a release run, so they appear in the report's Tier 1 section). In `MATRIX.md` the ambient profile's T0.14 re-check shares Tier 0's T0.14 row (the cell shows both verdicts, e.g. `PASS/PASS`), and a named profile's re-checks each get their own `[profile]` row. When multi-profiling, read each profile's overlay in `profiles.yaml` directly to confirm its window declaration (T0.4 is not re-run per profile).
### The Preflight Gate
Whenever a paid tier (1, 2, A or 3) is selected, four Tier 0 probes must not FAIL before any paid call: **T0.1** (route), **T0.3** (env coherence), **T0.16** (credential overlap) and **T0.17** (tier aliases; it SKIPs on the Anthropic routes and a Claude-family OpenRouter install). Each such FAIL means the paid calls would test, or bill, the wrong configuration.
- **Tier 0 selected:** its own results are reused; nothing runs twice.
- **Tier 0 not selected** (or it crashed before a gate probe ran): the missing gate probes run as the **gate subset**. Their names end in ` (gate subset)`, and the report opens its per-probe detail with a "Tier 0 (gate subset)" section.
- **Any gate FAIL:** every paid tier is skipped and `T0.ABORT` (FAIL) is recorded, its detail naming the blocking probe IDs. Tiers N and D still run. A crash while running the gate subset is a `T0.CRASH` FAIL and blocks too (fail closed).
- **With `--profiles` (the profile gate):** once the ambient gate passes (or is overridden), each named profile re-runs all four gate probes (T0.1, T0.3, T0.16, T0.17) on its own environment, ambient plus overlay, next to its T0.14 re-check (see the Tier 0 limitation above). The ambient gate judges only the environment the runner was launched with, while a `profiles.yaml` overlay can set exactly what these judge: three of the four shipped profiles set all four tier aliases, and an overlay that sets `ANTHROPIC_BASE_URL` or the shim lane controls changes route detection (T0.1). A FAIL skips that profile's paid tiers and records a `T0.ABORT` named "Paid tiers skipped (profile gate)" for it; its siblings still run when their own gate and T0.14 pass. A crash in the profile gate is a `T0.CRASH` FAIL for that profile and blocks it (fail closed). When the ambient gate blocks, no profile runs, so there is no profile gate.
- **`--allow-preflight-fail`:** the paid tiers run anyway (to test a deliberately broken configuration). It overrides the ambient gate and the profile gate alike: the FAIL stays in the report, no `T0.ABORT` is written, and the identity block's Preflight gate row says the gate was overridden.
- **Not part of the gate:** T0.0 (it only reports the image flavor) and the per-profile T0.14 re-check above, which still blocks just its own profile and is not overridden by `--allow-preflight-fail`. The cost confirmation that `--yes` skips is separate.
The pre-launch summary prints a `PREFLIGHT GATE` line naming the four probes whenever a paid tier is selected, plus a line on the per-profile re-check when named profiles are given. `report.json` records the outcome in `preflight_gate`, the profile gate in its `per_profile` key (see § Reading a Report).
---
## Tier 1 — One Live Round-Trip
A single cold-start `claude -p` call plus the plumbing checks around it. Three of these probes often report **INFO** on a healthy install: T1.4 and T1.5 because of short-headless-run mechanics, and T1.9 because the Tier 1 prompt forbids every tool but one Bash call. That is expected, not an install defect, and knowing why prevents false alarms.
**Live-run health gate.** Before judging anything, the suite checks whether a real model actually answered. A `claude -p` run can exit cleanly while no model ran. With an unknown model slug, an exhausted spend cap, or an API error, the CLI writes a local assistant record: `message.model` is `<synthetic>`, `isApiErrorMessage` is `true`, and `error` is `"model_not_found"` (observed 2026-09-24). That record's error sentence then comes back as the "response". The gate fails the run if any of these hold:
- the CLI result has `is_error: true`, a `subtype` other than `"success"` (except `"error_max_turns"`, below), or a non-null `api_error_status`. Every `api_error_status` seen so far has been null, so treating a non-null value as an API failure is inferred, and the reason says so;
- the transcript **ends** on an API-error record: some assistant record has `isApiErrorMessage: true` or an angle-bracketed placeholder model, and no real-model assistant record follows the last such record;
- `modelUsage` names no real model;
- the run timed out or the CLI errored.
T1.1, T1.7 and every Tier 2 probe then FAIL with that reason. T1.5 and T1.6 SKIP, since there is no real session to judge.
Two conditions are a **WARN** instead, because a real model did answer. In both cases the probe still judges its own evidence. A PASS becomes WARN, and its detail opens with "Live-run warning: …" (both conditions joined by "; " when both occur), ahead of the probe's own PASS reason, so the Problems table's *Observed* cell, which keeps the first 200 characters, shows why. A probe whose own verdict is WARN, FAIL, SKIP or INFO keeps its reason first, with the warning appended. This applies to T1.1, every Tier 2 probe, TA.1, T3.1 and T3.2, except a T2.7 guard FAIL: a guard that let the action through is recorded before the live-run checks, with neither a warning nor a live-run health row.
- **A recovered transient error.** A mid-response provider error is written as the same kind of `<synthetic>` record (`error: "server_error"`, text "API Error: Server error mid-response. The response above may be incomplete."). The CLI then retries. The observed pattern is real record → error record → real record, in about 20 headless transcripts on the GPT lanes. The warning gives the number of error records and the first one's error and text, truncated. If no real record follows the **last** error record, the run ended on the error, and that is a FAIL.
- **A `--max-turns` stop.** CLI result `subtype: "error_max_turns"`. The run hit its turn limit before finishing, so a missing completion token or marker FAILs on the probe's own check, with the max-turns note attached. The exact shape of this result object (for example whether `is_error` is also true) has not been observed in this repo, so the gate accepts either value.
| Probe | PASS means | Non-PASS interpretation |
|-------|-----------|-------------------------|
| **T1.1 Live round-trip response** | The live-run health gate is clean **and** the response contains `SMOKE_TIER1_COMPLETE`. Any response text alone is not enough. | WARN = the token came back, but the gate recorded a recovered transient error or a `--max-turns` stop (the detail says which). Re-run if it recurs. FAIL = the gate tripped (the detail names the signal, e.g. `transcript API-error record (error='model_not_found')`), or a real model answered without the completion token. A real routing, auth or model-selection failure surfaces here first. Check the provider-specific failure modes below, and T0.12 for a stale local `ANTHROPIC_MODEL` or `ANTHROPIC_DEFAULT_MODEL`. |
| **T1.2 Transcript located** | The session transcript was found (archived or live). | FAIL = transcript missing in both locations; session logging may be broken. |
| **T1.3 Audit-log hook fired** | `audit.jsonl` has entries for the session. | FAIL = the audit-log hook did not fire; check its registration (T0.6). |
| **T1.4 Context-reporter injection** | A **real** injection appeared: `Context utilization [` inside a hook attachment (`hook_additional_context` / `hook_success`). Each injection is recorded as both of those attachments with the same `toolUseID`, so the count is deduplicated by `toolUseID` (one injection counts once). CLAUDE.md and skill bodies describe the reporter in prose; that prose never counts, because it is in every session: a bare substring match would PASS a session with no injection at all. | **INFO when absent** is expected for a short headless probe. The first hook call in a fresh session isn't rate-limited. But `context-reporter.sh` emits nothing until it can measure the transcript, and later calls are limited to one per 60 seconds. A one-tool probe often ends before either allows an injection. On 2026-09-24 both Tier 1 runs had none, while a longer Tier 2 dispatch run had one. Absence is not evidence the hook is broken: T0.6 covers its registration and `tests/bash/context_reporter.bats` covers its behavior. |
| **T1.5 /tmp coordination caches** | The `/tmp/claude-model-{session}` cache (written by `context-reporter.sh`'s `resolve_model_cache`) **equals the model the transcript actually ran**. The read uses a bounded ~5s retry and quotes both the first and final read. | **WARN** = the cache names a different model than the transcript, so something other than this session's hook pass wrote it. **INFO** = the model cache is absent; the hook writes it only once the transcript already records an assistant model, and a one-tool run can finish first. **SKIP** = the live-run gate tripped. The `/tmp/claude-ctx-window-{session}` cache never affects the verdict. Its only writer is the `context-bar.sh` statusline, and `claude -p` renders none, so an absent window cache is expected. T1.5 runs **before** T1.6, and T1.6 writes only report-local caches, so T1.5 never reads back the harness's own synthetic values (T1.6's `200000` window is the harness's payload constant, not headless Claude Code behavior). |
| **T1.6 Statusline against real session** | `context-bar.sh` renders non-empty output against the real session transcript. The payload model is the model observed in the transcript (falling back to `modelUsage`, then the configured id). `DAAF_CONTEXT_BAR_CACHE_DIR` points at the report's `evidence/t16_statusline_cache/`, so the render cannot write the session's `/tmp` caches. The payload's `200000` window is a synthetic value: this probe proves the renderer handles the real transcript, not the session's window. | WARN = no output or a timeout. SKIP = no transcript, or the live-run gate tripped. |
| **T1.7 Token/cost parsing** | A positive input or output token count parsed from the JSON result's `modelUsage`. | **FAIL** = the live-run gate tripped. Zero or missing usage on such a run is the failure, stated with its reason; it is not treated as a parse question. WARN = a real run whose token fields are zero or `None` (the parse may have missed the `modelUsage` block). `None` token fields (a timeout or CLI-error run) are judged by these same rules and never abort the run. On `anthropic-api` the detail adds "Cost is real spend on this route", with the `--max-budget-usd` cap in the evidence. |
| **T1.8 Served identity** | The run uses `--output-format stream-json`, and the allowlisted `system/init` record (`apiKeySource`, `model`, `permissionMode`, version) matches the route: `apiKeySource` `none` on `anthropic-subscription`, a paid source (`ANTHROPIC_API_KEY`, `/login managed key`, `apiKeyHelper`) on `anthropic-api`; the init and transcript models are Claude-family on Anthropic routes and GPT-family on shim routes (OpenRouter: any). | **FAIL** = a paid key source on the subscription route (requests bill a key), `none` on the API route (no key in use), or a served model of the wrong family. **WARN** = no init record captured, or an unrecognized `apiKeySource`. On `anthropic-api` a budget-cap stop also shows up here and in the live-run gate. Fix with T0.16 (credentials) or T0.14/T0.17 (model mapping). |
| **T1.9 Orchestrator bootstrap** | The T1 transcript holds the `UserPromptSubmit` hook attachment carrying the reminder text ("invoke the daaf-orchestrator skill") **and** a non-error `Skill` call for `daaf-orchestrator`. **INFO** = the reminder fired and the skill was not loaded. That is the **normal** result on a healthy install: the T1 prompt allows one Bash call and "not … any other tools", and the model usually obeys the prompt over the reminder. The hook firing is the deployment fact this probe proves; skill loading is T2.4's check, and the orchestrator loading on a real question is T3.1's. | **WARN** = the skill loaded but no reminder attachment was recorded: the attachment shape may have changed, so read the transcript. **FAIL** = neither the reminder nor the skill appears, or the live-run gate tripped: check `remind-orchestrator.sh` registration for `UserPromptSubmit` (T0.6) and skill loading (T0.23). |
| **T1.10 Session archive** | Within about 10 s of the run, `archive-session.sh` (SessionEnd) wrote both `.claude/logs/sessions/*_<sid8>_orchestrator.jsonl` and `.md`. | **WARN** = only one of the two files. **FAIL** = neither: check the SessionEnd registration and that `.claude/logs/sessions/` is writable. |
**Why the headless caveats matter.** `claude -p` renders no statusline, so nothing in a headless session writes the context-window cache. And `context-reporter.sh` can only write the model cache or inject after the transcript holds something to measure. The suite therefore judges only what a headless run can positively show: the model cache must match the transcript model when present, and a hook injection counts only when it is real. Absence is reported as INFO with its reason, not as PASS or WARN.
---
## Tier 2 — Eight-Probe Functional Battery
Each probe is a separate cold-start run. All checks are capability-structural and deliberately tolerant of stylistic/protocol variation. Each probe first applies the Tier 1 **live-run health gate**. A run where no real model answered (a transcript that ends on a `<synthetic>`/API-error stub, `is_error`, a non-`success` subtype, a non-null `api_error_status`, or an empty `modelUsage`) is a **FAIL** that names the reason, never a PASS or a vacuous SKIP. A recovered transient error, or a stop at `--max-turns` (subtype `error_max_turns`), is not a gate FAIL. The probe judges its own evidence, and a PASS becomes WARN with the reason (see the Tier 1 gate above).
Every Tier 2 run works inside its own **per-run, UUID-owned sandbox** — `scripts/deploy_smoke/_sandbox/run_<uuid>/`, created fresh at the start of the run and removed in a guaranteed cleanup path (scoped to exactly that directory — never a recursive wipe of `_sandbox/` and never a sibling). This is what makes the T2.1, T2.2 and T2.8 fixtures fresh *by construction*: a run directory left behind by a hard process kill carries a UUID no future run mints, so it is inert crash residue that cannot satisfy a later run. Probes read only their own `run_<uuid>/` directory, so a file at the `_sandbox/` root itself (such as `t21_marker.txt` or `t22_probe.py`) is never evidence; all of `_sandbox/` stays git-ignored.
| Probe | PASS means | FAIL/SKIP interpretation |
|-------|-----------|--------------------------|
| **T2.1 Subagent dispatch + search** | An `Agent`/`Task` tool_use occurred **and** the marker token came back in the orchestrator's response. | WARN = a dispatch happened and a subagent transcript exists, but the marker did not come back: dispatch works, relaying the result does not. FAIL = anything else (no dispatch, or a dispatch with no subagent transcript and no marker). On non-Claude routes, cross-check the ceiling-hook posture (T0.2) — a mis-set remap can block dispatch. **Tier routing:** the evidence also compares the subagent's served model with the alias mapping for the agent's `model:` tier on this route (`ANTHROPIC_DEFAULT_<TIER>_MODEL` when set; otherwise a Claude model whose name contains the tier). A mismatch turns a PASS into **WARN** ("Tier routing mismatch: …"); fix that mapping. An unobserved model or tier leaves the verdict alone (`routing_ok=None` in the evidence). |
| **T2.2 Coding agent write + execute** | `research-executor` wrote a script and ran it via `run_with_capture.sh`, and *this run's* execution left fresh captured evidence. PASS requires **all** of: this run's probe script exists; **exactly one** of the wrapper's three-line execution-log banner *frames* is present, where a frame is `# ` + **40 or more** `=` / `# EXECUTION LOG` / the same rule again (the wrapper writes 77 `=`); a `# Exit code: 0` record and a `# Capture: COMPLETE` record both appear **after** that banner; the wrapper's structural records `# Command: …`, `# --- STDOUT ---`, and `# --- STDERR ---` all appear after the banner **in that order** (fact `frame_records`); and this run's unique nonce appears **between** the `# --- STDOUT ---` and `# --- STDERR ---` markers (fact `nonce_in_stdout`) — the only region of the log carrying `# `-prefixed captured output. The nonce (not just the script's presence) is what ties the PASS to freshly captured output rather than source text; the rule-width floor and the record skeleton are what keep an agent from forging the evidence in the script source. | FAIL = any one of those is missing. A nonce that appears only in the script source *or* only outside the captured-STDOUT region, a stale banner from a prior run, a recorded nonzero exit, or a `Capture: INCOMPLETE` record (the wrapper's signal that the archived stdout may be truncated, exit 74) are each a **FAIL** — never a PASS-with-note. So are both banner-count failures, reported as `banner_count`: **0** = no qualifying frame — a bare `# EXECUTION LOG` line written into the source, *or* one wrapped in a token `# =` rule, is not a banner (the script never ran); **2+** = a forged frame stacked ahead of the real appended log, or a duplicated append (ambiguous evidence, so FAIL even if the records after the first frame look clean). The width floor matters because the wrapper **refuses to run** a script whose source already carries the marker: a source-forged frame is never joined by a real one, so `banner_count` stays at the passing value of 1 and the frame itself must be rejected on its shape. Confirms the file-first execution path and `enforce-file-first.sh` are intact *for this run*, not merely that a script file exists. |
| **T2.3 Web access (WebSearch + fetch protocol)** | A web call in the main or a subagent transcript returned a tool_result **without `is_error`**. A web call is a `WebSearch`, or a `Bash` call running `web_fetch.sh` (the DAAF fetch protocol). The built-in `WebFetch` is NOT a pass signal: the `block-webfetch.sh` hook blocks it, so its presence means the agent used a disabled path, not working web access. | FAIL = no such call, or every web call errored or recorded no result (the evidence counts `web_calls`, `succeeded` and `errored_or_no_result`). Possible causes: a provider that does not surface WebSearch, a pre-rebuild image missing the fetch tool's dependencies, or a sandboxed network. **WARN (shim routes only)** = the pass rests only on the `web_fetch.sh` fallback: the WebSearch tool itself returned no non-error result, which is the shim's hosted search failing (F25; the evidence counts `websearch_succeeded`). Read the subagent's WebSearch result and the shim log. |
| **T2.4 Skill loading** | A `Skill` call for **`polars`** (bare, or plugin-qualified `x:polars`) returned a non-error tool_result. Any Skill call is not enough, because the orchestrator skill loads first in most sessions. | FAIL = no successful `polars` Skill call; the evidence lists every skill that was called. Progressive disclosure may not be firing, or the model loaded the wrong skill. |
| **T2.5 Isolation-strip hook** | An `Agent`/`Task` tool_use asked for `isolation`, and `block-remote-isolation.sh`'s own hook attachment (its `command` names the hook) carries `updatedInput`. | **SKIP is tolerated**: if the model omitted or refused the `isolation` parameter, there is nothing to strip (not a FAIL). WARN = isolation was requested but that hook attachment is absent, though the dispatch did not hang. Another hook's `updatedInput` does not count. |
| **T2.6 Nested-dispatch prevention** | The dispatched subagent could not launch a nested subagent. The verdict reads **only the subagent transcripts**, never the main transcript's content (the main transcript is read only to link each subagent transcript to its top-level call), because the orchestrator skill's prose about nested subagents there would read as a "denial". There are two PASS layers. **(a) Harness layer:** the subagent attempted nothing, and neither its recorded tool list (the `prompt_snapshot` attachment's `tools`) nor its deferred tool names (`deferred_tools_delta` attachments' `addedNames` minus `removedNames`, when present) include `Agent` or `Task`. Names match exactly, so the deferred `TaskOutput`/`TaskStop` tools don't count. On 2.1.281 the attachment is sometimes absent and sometimes present. The 17:29 live run's subagent on 2026-09-24 listed 22 deferred names, including `TaskStop` but not `Agent` or `Task`. With `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH=1` in `settings.json` `env`, Claude Code withholds both tools from every subagent (observed 2.1.261 and 2.1.281; the Claude Code changelog says only that the setting "disable[s] nesting"). The detail then reads: "Nested dispatch unavailable at spawn depth 1 (harness layer): the subagent's tool list (including any deferred tool names) has no Agent or Task tool, so it could not attempt one. Hook backstop (block-nested-dispatch.sh) not exercised." This is the expected result at spawn depth 1. The harness layer also PASSes when the subagent **did** try a nested `Agent`/`Task` call while the tool was absent, and every such attempt got an error tool_result. The live 2026-09-24 run (session b5524e7f) showed this: the subagent called `Agent` anyway and got `<tool_use_error>Error: No such tool available: Agent. Agent is disabled for this session, in subagents as well as here.</tool_use_error>`. The detail then reads: "Nested dispatch rejected at spawn depth 1 (harness layer): the subagent's tool list (including any deferred tool names) has no Agent or Task tool, and each nested Agent/Task attempt got an error result because the tool was absent. Hook backstop (block-nested-dispatch.sh) not exercised." That wording is used only when every error text contains "No such tool available". Any other error text gets the neutral form: "Nested dispatch rejected at spawn depth 1 (harness layer): every nested Agent/Task attempt returned an error while Agent/Task was absent from the subagent's tool list (including any deferred tool names). Hook backstop (block-nested-dispatch.sh) not exercised." Tool absence counts only when **every** subagent transcript recorded its tool list; if any list is unrecorded, availability is unknown and an attempt grades WARN. **(b) Hook layer:** the subagent attempted an `Agent`/`Task` call and its transcript holds a real deny record. That is either an `is_error` tool_result (observed on earlier pins) or a hook attachment carrying `block-nested-dispatch.sh`'s text "may not launch nested subagents" (inferred shape, not yet observed live). At depth 1, `block-nested-dispatch.sh` is a backstop for when the limit is lifted or the harness regresses; its deterministic coverage is `tests/bash/block_nested_dispatch.bats` in Tier D. | **FAIL** = a nested dispatch actually ran. Three signals show it. (1) A subagent's own `Agent`/`Task` call got a non-error tool_result, wherever the child's transcript landed. (2) A subagent transcript links to no top-level `Agent`/`Task` call in the main transcript. Each transcript is linked to its parent by its `agentId`, which equals the parent result's `toolUseResult.agentId`, or by its opening prompt, which equals the parent's `input.prompt`. The prompt match was checked on stored transcripts: 10/10 subagents of one session, plus a terminated-early subagent and the 2.1.281 live one. (3) As a backstop for transcripts with no identity, there are more subagent transcripts than top-level calls. Errored top-level calls stay in the count. An "Agent terminated early due to an API error" result **did** start a subagent, and that subagent left a transcript; its result has no `agentId`, so its transcript links by prompt. A retry after such a failure (two calls, two transcripts) is therefore not read as nesting. The evidence records `top_level_dispatches`, `top_level_errored`, `linked_subagents`, `unlinked_subagents`, `unlinkable_subagents`, `retried_dispatch`, `nested_calls`, `nested_errored`, `nested_succeeded` and `nested_error_text`. If the main transcript is missing, nothing can be linked, and the check assumes one outer dispatch. Linkage knows only the main transcript's `Agent`/`Task` calls. A subagent started any other way inside the T2.6 session therefore reads as unlinked, and so as nesting (FAIL), although nothing nested. Examples are a slash-command review such as `/code-review` and a forked skill. Future probe changes must not launch such subagents inside the T2.6 session. On a FAIL, check that `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH=1` is still in `settings.json` `env`, and that the hook sits first in *both* the `Task` and `Agent` matcher chains and still receives the `agent_id`/`agent_type` fields it keys on. **WARN** = a nested attempt with no spawn and no deny record that is not the harness rejection above, e.g. the tool was available, or an attempt has no recorded result (inconclusive). **INFO** = no attempt and no recorded tool list, so only the model's own report is available, which is not evidence. **SKIP** = no subagent was dispatched (see T2.1), or the subagent had `Agent`/`Task` (in its tool list or as a deferred tool) but ignored the instruction. |
| **T2.7 Guardrails live** | One session asks the model to run exactly `pip install requests` with Bash, then to Read `/daaf/.env`. Every `pip install` call and every `.env` Read came back as an error in a **recognized deny shape**, matched exactly: the hook text `BLOCKED by bash-safety hook`, or Claude Code's permission-rule phrase for that tool, "Permission to use Bash with command … has been denied" (pip half) or "Permission to read … has been denied" / "Permission to use Read … has been denied" / "File is in a directory that is denied by your permission settings." (read half). The Bash phrase was observed in a stored subagent transcript on 2026-09-29; the first two Read phrases come from the 2.1.261 CLI source, and the third was found by a reviewer in the installed 2.1.281 binary (Read's input validation when a deny rule matches the path, 2026-10-13). None is confirmed by a live T2.7 run yet. A Bash call counts as a pip attempt only when one of its command segments (split on `;`, `&`, `\|`, parentheses and newlines outside quotes) actually invokes pip install: `pip`/`pip3`, optionally after `sudo`, `VAR=value` assignments, `python3 -m` or `uv`. A shell wrapper's `-c` string (`bash -c "pip install x"`, `sh -lc '…'`, `bash --login -c …`, optionally behind `sudo`/`env`/`exec`/`command`/`nohup`/`timeout`/`nice`, whose own option arguments such as `sudo -u root` or `timeout 30` are skipped) is unwrapped and matched the same way, bounded to 3 levels; the same prefixes are skipped before a direct `pip install`. A `grep -n "pip install" …`, an `echo` or a `--help` that only mentions the phrase is not an attempt, so its successful result is no FAIL. The Read is requested only when no real `/daaf/.env` exists, so a Read that reaches the filesystem errors ("File does not exist"): an error alone is not a deny. When a real `/daaf/.env` exists, the read half is not requested and the pip half is judged alone (the detail says so). This proves the PreToolUse hooks and the deny rules fire for this route's tool-call shape. | **FAIL** (pip) = a `pip install` call returned a non-error result, or an error whose text is pip's own output (`Collecting …`, `Requirement already satisfied`, `Successfully installed`, `Could not install packages`, `Defaulting to user installation`, `Downloading ….whl`; this wording is inferred from pip's messages, not a stored transcript): pip ran, so the install guard did not fire. **FAIL** (read) = the `.env` Read returned a non-error result, or an error without deny text (for example "File does not exist"): the Read was attempted, so the `Read(.env)` deny rule did not fire. Either FAIL is decided even if the live-run gate also complains. Check `bash-safety.sh` §8 registration for Bash and the `Bash(pip install*)` / `Read(.env)` deny rules in `settings.json`. **WARN** = an error with deny-like words ("denied", "not permitted", "blocked", "permission") in no recognized shape, or a pip error with unexplained text; the detail quotes up to two of them. Nothing ran, but the wording is new: read the transcript, and if it is a real deny, add its shape to `smoke_probes._PERMISSION_DENY_BASH_RE` / `_PERMISSION_DENY_READ_RE`. Also **WARN** = the model did not attempt both calls (or the pip call when the read half is off), or an attempt has no recorded tool result (the detail says either). The evidence counts `pip_attempts`, `pip_denied`, `pip_ran`, `pip_unrecognized`, `read_attempts`, `read_denied`, `read_ran`, `read_unrecognized` and `bash_safety_text_seen`. |
| **T2.8 R coding agent write + execute** | The R twin of T2.2: `research-executor` writes `t28_probe.R` in the same per-run sandbox, printing this run's nonce (`daaf-exec-r-` plus 12 hex characters) with `cat()`, and runs it through `run_with_capture.sh`. It is judged by the same evidence rules as T2.2 (the shared judge: exactly one framed banner, `# Exit code: 0` and `# Capture: COMPLETE` after it, the record skeleton in order, the nonce inside the captured-STDOUT region). This exercises the R branch of `run_with_capture.sh` (Rscript) and the Rscript rule of `enforce-file-first.sh` live, which otherwise only the bats suites cover. | The same FAIL readings as T2.2. Read the R probe script's appended execution log, then check `run_with_capture.sh`'s R branch, `enforce-file-first.sh` and the `research-executor` agent. T0.11 (R locale) and T0.22 (`Rscript` present) cover the R install itself, so read them first when every R run fails. |
---
## Tier D — Deterministic Battery (zero API cost)
Selected with `D` in `--tiers` (with `--include-r-smoke` for TD.6, `--tier-d-all` for TD.7 off the shim routes). It runs after every other tier. Every probe shells a repo test entry point and reports its exit code — no LLM, no cost. A missing tool is a **SKIP**, not a FAIL, so Tier D runs on the standard image too: there TD.1 and TD.2 SKIP, because only the developer image (`DAAF_DEV=1` at build time) installs bats and PowerShell (T0.0 reports the flavor). TD.9 always runs, as TD.8 does.
**The harness self-tests itself first.** TD.0 discovers `tests/python/test_deploy_smoke*.py` *before* the broader batteries. This includes the existing harness contracts plus shared main-model preflight. These are provider-free contract tests, not live provider acceptance. For agent execution, use a fresh flat audited launcher with `unittest.defaultTestLoader.discover('/daaf/tests/python', pattern='test_deploy_smoke*.py')`, `sys.dont_write_bytecode=True`, and an assertion on `TextTestRunner(...).run(suite).wasSuccessful()`, executed through `run_with_capture.sh`.
**Deterministic environment isolation.** All Tier D subprocesses run under a sanitized copy of the live environment with exactly two variables stripped — `CLAUDE_CODE_MAX_CONTEXT_TOKENS` and `DAAF_BRANCH` — because a fully-configured live install legitimately exports both, and both steer default-window / default-branch fixtures onto the live session's values. This is defense-in-depth *atop* each battery's own fixture isolation (bats `setup()` unset, Pester `BeforeEach` removal), **not** `env -i`: `PATH`, `HOME`, route credentials, and the developer toolchain are all preserved so the batteries can still find their interpreters. TD.0's evidence records which variables were actually removed.
**Two-level evidence policy.** On a **PASS**, a probe keeps only a concise final excerpt (the last several lines) in the report. On a **FAIL or timeout**, the report quotes a bounded head-and-tail excerpt (so both the earliest failure names and the final summary stay visible) *and* the COMPLETE scrubbed output is persisted to `evidence/tier_d/{probe_id}.log`, referenced from the probe's evidence. All captured output is scrubbed for secret env values before it is quoted or persisted.
| Probe | PASS means | FAIL/SKIP interpretation |
|-------|-----------|--------------------------|
| **TD.0 harness self-test** | The deploy-smoke suite's provider-free `test_deploy_smoke*.py` modules pass — the harness's env sanitization, evidence capture, output routing, Tier 2 cleanup, and T2.2 freshness logic are all intact. Runs first so the rest of Tier D is trustworthy. | FAIL = a harness regression: fix the harness before trusting any other Tier D verdict. SKIP is not expected (it only needs stdlib `python3`); a FAIL here means the suite's own contracts broke. |
| **TD.1 bats tests/bash** | `bats tests/bash/` exits 0 (all Bash unit tests pass). Its timeout is at least 1800 s whatever `--timeout` says (`smoke_probes.TD1_MIN_TIMEOUT_S`; a larger `--timeout` wins). | FAIL = a Bash test failed. SKIP = `bats` not installed (the standard image; only the developer image carries it). **Known issue, suite size against the timeout:** the battery grows with every bats file (3287 tests on 2026-11-01) and ran past the 600 s Tier D default in that day's live run, so a FAIL whose detail reads `bats tests/bash timed out after …s` with no failing test in `evidence/tier_d/TD.1.log` is the clock, not a regression. The 1800 s floor is headroom over that observation; the full-suite time has not been measured. Pass a larger `--timeout`, or run `bats tests/bash/` by hand. The report's hint says which case it is: on a timeout it gives these two remedies, on any other FAIL it points at the failing test in that log. |
| **TD.2 Pester tests/powershell** | `Invoke-Pester -Path tests/powershell -CI` exits 0 **and** its NUnit `testResults.xml` is present in the report at `evidence/tier_d/testResults.xml`. Pester runs with its working directory set to that evidence dir, so the XML lands report-local rather than at the repository root (`/daaf/testResults.xml` is never created). | FAIL = a PowerShell test failed, **or** Pester exited 0 but the report-local `testResults.xml` is missing (an artifact-routing failure — a zero exit with no XML is a TD.2 FAIL by contract, not a PASS). SKIP = `pwsh` not installed (the standard image; only the developer image carries it). |
| **TD.3 daaf-conventions lint** | `tests/lint/check-daaf-conventions.sh` exits 0. | FAIL = a convention violation. Read the tail evidence for the offending file. |
| **TD.4 safety-hook tests** | `scripts/test_safety_hooks.sh` exits 0 (the `bash-safety.sh` battery passes). | FAIL = a safety-hook regression — investigate before trusting the deployment's guardrails. |
| **TD.5 single-command hook tests** | `scripts/test_enforce_single_command.sh` exits 0. | FAIL = an `enforce-single-command.sh` regression. |
| **TD.6 R/Python skill smoke suite** | (opt-in) log-stripped copies of `scripts/smoke_tests/smoke_*` run clean via `SMOKE_DIR`. | FAIL = a library smoke failed. **SKIP by default** — pass `--include-r-smoke` to enable (it is slow). Mirrors the CI staging pattern; runs under `scripts/scratch/smoke_live/`, never `/tmp`. |
| **TD.7 provider-shim unit suite** | The CI command `python3 -m unittest discover -s tests/provider_shim -t tests -p 'test_*.py'` exits 0 (timeout at least 900 s). It tests the shim source on disk, not the running daemon; T0.8 compares those. Covers most shim failure-catalogue rows (see the map below). | FAIL = a shim test failed: read `evidence/tier_d/TD.7.log`; CI's provider-shim job runs the same command, and also runs `scripts/provider_shim/probe_context_ceiling.py --self-test`, which TD.7 does not. **SKIP** off the shim routes unless `--tier-d-all`. |
| **TD.8 core library imports** | A log-stripped copy of the latest `scripts/smoke_tests/smoke_imports*.py` revision, staged under `scripts/scratch/smoke_live_td8/`, runs clean. Always on, and fast; TD.6 stays the slow opt-in suite. | FAIL = a pinned library failed to import (rebuild the image; read `evidence/tier_d/TD.8.log`), no `smoke_imports*.py` exists, or staging failed. |
| **TD.9 Local browser services** | For **marimo** and **code-server**: the binary starts on a free port bound to `127.0.0.1` and answers HTTP (any status, redirects not followed, no proxy) within 45 s, then its process group stops (SIGTERM, then SIGKILL after 10 s). Each child runs offline (update checks and telemetry off) with `HOME`, the XDG folders and `TMPDIR` pointed at `evidence/tier_d/td9/`, so it never touches the maintainer's real marimo or code-server state or `/tmp`. Each child also gets no credential-named variable: any name the report redaction treats as secret (`KEY`, `TOKEN`, `SECRET` or `AUTH` in it) or the package scrubber's secret-name vocabulary matches (`PASSWORD`, `COOKIE`, `CREDENTIALS` and the like) is withheld, and the run evidence counts them ("N credential-named env var(s) withheld from the service"); `PATH`, `LANG` and `LC_*` are kept. marimo gets a throwaway access token, generated for the run, deleted after it and scrubbed from the evidence as `<redacted:td9-token>`, because its terminal websocket is guarded only by that token; code-server runs with `--auth none` on loopback. A static check confirms `scripts/launch_marimo.sh` and `scripts/launch_code_server.sh` pass `bash -n` and still call their binaries by name. The detail says "invoke the tested binaries" only when both services ran; when a binary was absent it says "still call their binaries" and names what was and was not tested. Always runs, like TD.8. | **FAIL** = a present binary gave no HTTP answer in time, exited early (an early exit that reports the address in use, where another process took the chosen port first, is retried once on a fresh port and noted in the evidence) or could not be stopped, or a launcher fails `bash -n` or does not call its binary. The evidence is the last 40 scrubbed lines of the service's own output, in `evidence/tier_d/td9/<service>.log`. **SKIP** per absent binary; the probe SKIPs as a whole only when both are absent. **Trade-off:** TD.9 does not run the launchers themselves, because they use fixed ports (2718 for marimo) and could collide with, or take over, a service the maintainer has running; a configuration bug inside a launcher is therefore not caught live. **Not covered:** the log viewer. `log_viewer_server.py` takes `--port` but binds all interfaces, so TD.9 leaves it to `tests/python/test_log_viewer_server.py`. |
---
## Tier N — Network and Data Services (free, no model calls)
Runs once, after Tier 0. Only TN.1 can FAIL; the others WARN, because a flaky third-party service should not fail an install. Artifacts land under `evidence/tier_n/`; fetched content is never quoted (it is untrusted).
| Probe | PASS means | Non-PASS interpretation |
|-------|-----------|-------------------------|
| **TN.1 Route endpoint reachability** | A credential-free HEAD (no redirects followed) reaches the route's endpoint over TLS: `api.anthropic.com` (both Anthropic routes), `chatgpt.com`, `api.openai.com/v1/models` or `openrouter.ai/api/v1/models`. On shim routes the shim's `/health` must also answer. Any HTTP status, including 401/403/404, proves reachability. | **FAIL** = the endpoint or the shim is unreachable: check network and proxy settings, and start the shim on shim routes. |
| **TN.2 web_fetch.sh round-trip** | `web_fetch.sh https://example.com/ evidence/tier_n/tn2_web_fetch` exits 0 and leaves an extract and a MANIFEST row. | **WARN** = it did not complete; run it by hand and read its exit code against the `web-retrieval` skill's table. |
| **TN.3 scholar_search verify** | `scholar_search.sh verify 10.1038/nature14539` (a free OpenAlex call) exits 0 with a `CONFIDENT`, `WEAK` or `EXISTS` verdict. Only the exit code and verdict word are kept. | **WARN** = unresolved: run it by hand (the `scholarly-search` skill's exit table; 4 and 6 are network errors). |
| **TN.4 Education-data mirror** | A HEAD of `saipe/districts_saipe` through the first mirror in `education-data-query/references/mirrors.yaml`, at its pinned revision, returns a status below 400. | **WARN** = unreachable or an error status, or `mirrors.yaml` could not be read: check the pinned revision and access to the mirror host. |
---
## Tier A — Auto Mode (opt-in, 1 call)
| Probe | PASS means | Non-PASS interpretation |
|-------|-----------|-------------------------|
| **TA.1 Auto mode (harmless command)** | One `claude -p --permission-mode auto` call is asked to run exactly `openssl dgst -sha256 /daaf/CLAUDE.md`. The Bash call ran without error and the reply carries the SHA-256 the harness computed itself. On shim routes, the shim log for that call's time window also shows at least one successful classifier-style terminal: an `event=terminal` line with `outcome=success`, ` stream=n ` and ` tools=0 `. TA.1 runs its own counts-only scan of that window for this (internally named `TA.1-scan`; it is not a separate report row, and only its `classifier_terminals` count reaches TA.1's evidence). **The signature is unconfirmed until the first live run:** nothing in the shim's terminal record marks a request as the Auto-mode classifier, and Claude Code's other non-streaming, tool-less calls (session titles, resume summaries) match it too. A PASS therefore shows that some such call succeeded in the window, not that the classifier did. | **FAIL** = the command was blocked, refused or not attempted; the digest is missing from the reply; or, on a shim route, no classifier-style terminal was logged (Auto mode's classifier path may be broken; F12, F13). Evidence in the package: `logs/shim.log` (the run window). Like every live probe, TA.1 first applies the Tier 1 **live-run health gate**: a run where no real model answered FAILs with that reason, and a recovered transient error or a `--max-turns` stop turns a PASS into WARN. **WARN** = on a shim route the log could not be scanned. A shim restart inside the call window does not reset the classifier count. This is the gate before moving the Claude Code pin past 2.1.281 (see Auto mode below). |
## Tier 3 — DAAF Golden Path (opt-in, 2 calls)
| Probe | PASS means | Non-PASS interpretation |
|-------|-----------|-------------------------|
| **T3.1 Orchestrator engaged** | `claude -p "What does grade = -1 mean in CCD enrollment data?"`, with no mode confirmed: the session's main transcript holds a `Skill` load of `daaf-orchestrator` (bare, `/name` or `plugin:name`) with a recorded non-error tool_result. Gate adherence (does the reply name a mode and ask before answering) is DAAFBench's to grade, so the evidence records it as information only, and it never changes the verdict: `mode_named` (one of the ten DAAF engagement modes: Data Onboarding, Data Lookup, Literature Review, Data Discovery, Ad Hoc Collaboration, Full Pipeline, Revision and Extension, Reproducibility Verification, Framework Development, User Support; case-insensitive, a space or hyphen between words; the bare word "mode" does not count), `asks_confirmation` (the last ~400 characters ask a confirmation question: a confirm phrase (confirm, proceed, shall I, should I, would you like, do you want, want me to, go ahead, sound(s) good, ok(ay) to, does that work, is that ok/right/correct/what you want) followed by a `?` within the same sentence; a bare `?` does not count) and `prek_already_answered` (the reply mentions Pre-K anywhere; this also matches a reply that names Pre-K only in a scope example without answering, as in the 2026-11-02 run below). The evidence also records `orchestrator_skill_loaded` (true, false, or null when the main transcript is missing, empty or unreadable). The detail gives the verdict reason first, then "Recorded, not graded: mode named: …; asks confirmation: …; Pre-K mentioned: …". | **WARN** = `daaf-orchestrator` did not load: no non-error `Skill` call for it in the main transcript. Not a FAIL, because the structural causes are FAILs elsewhere (T0.6 for the `remind-orchestrator.sh` registration, T0.23 for the skill's frontmatter, T1.9 for the reminder firing); with those clean, a no-load in a real session is most likely model behaviour. Read the main transcript (`transcripts/<sid8>/main.jsonl` in the package). Also **WARN** when the session's main transcript is missing, empty or unreadable, so the load could not be checked (the detail opens with "daaf-orchestrator load not checked: …"; `orchestrator_skill_loaded` is null): look at where the transcript should be, as for T1.2. The Tier 1 **live-run health gate** applies first: a run where no real model answered is a FAIL with that reason (and T3.2 then SKIPs); a recovered transient error or a `--max-turns` stop turns a PASS into WARN. |
| **T3.2 Golden path (resume + skill retrieval)** | The same session is resumed with `--resume <session>` (not `-c`, which picks the most recent conversation in the folder) and the prompt "Yes, go ahead.", and three checks hold. **The resume continued the T3.1 session:** the CLI's own result does not report another session id, and the session's main transcript holds a user record whose text is "Yes, go ahead.". In print mode `--resume` keeps the session id and appends the new turn to the T3.1 transcript file (observed 2026-11-02, Claude Code 2.1.281). Only a different reported id is evidence from the CLI: when the CLI reports no id, the harness fills in the resumed id, so the transcript check decides. The id the harness itself stores is no evidence either, because it is the id it asked to resume. **The answer says grade -1 means Pre-K**, a fact both `education-data-context` and `education-data-source-ccd` state. **The session shows an education-data skill retrieval:** a `Skill` load of any skill whose name starts with `education-data-` (bare, `/name` or `plugin:name`), or a `Read` of any file under a `skills/education-data-*/` folder (its `SKILL.md` or a `references/` page; absolute or relative, once `..`, `.` and repeated `/` segments are resolved; `skills` must be a whole path segment, so a lookalike such as `myskills/` does not count), with a recorded non-error tool_result. A call with no recorded result or an errored one does not count. Only when no transcript of the session holds such a `Skill` load or `Read` does a **Bash read** count: a `Bash` call whose command names a path under a `skills/education-data-*/` folder (a file or the folder itself, absolute or relative, resolved as for a `Read`, except that `skills` may also follow `=`, as in `--file=skills/…`; `skills` directly after `:`, as in `PATH=$PATH:skills/…` or `scp host:skills/…`, does not count, because a path in that position is mostly a path-list entry, a remote copy or plain text, while `git show HEAD:.claude/skills/…` counts through its `/`; the first such path in the command, a path ending at whitespace, a quote, a backtick or one of `| ; & ) < >`, with no shell parsing) and whose recorded result is non-error with non-empty text. `(Bash completed with no output)`, the text Claude Code records for a command that printed nothing (observed 2026-10-30, Claude Code 2.1.281), counts as empty, as does a null body. A Bash read is narrower than a `Skill` load or `Read`: only the command's output reaches the model, which for a `grep` is the matching lines. The rule matches the path in the command and does not parse the shell, so a listing of a skill folder (`ls`, `find`) and a `grep` whose pattern spells a skill path but which searches other files also count, though no curated text reaches the model; read the transcript when the signal is a Bash read. It counts toward PASS like the others, and the detail says "by a Bash read". A PASS proves the Pre-K answer came with a recorded retrieval, not that the answer rests on it: `education-data-context`'s description, in every session's skill listing, already says grade=-1 is Pre-K. In Data Lookup (the mode the orchestrator picked for this question in both 2026-11-01 and 2026-11-02 runs on `anthropic-subscription`) the orchestrator hands the lookup to a specialist, `search-agent` by default or `source-researcher` for a deep lookup, which loads the data skill in its own transcript. So the probe scans the main transcript and every subagent transcript of the session, whatever the agent type (the live `<session>/subagents/` folder, or the SessionEnd archive copies in `.claude/logs/sessions/` when that folder has none), read once after the T3.2 call. T3.1 and T3.2 share the session, so a retrieval counts whichever call it happened in. The evidence records `resumed_same_session` (true, false, or null when the main transcript is missing, empty or unreadable), `prek_answer`, `retrieval_seen`, `retrieval_signal` (what matched, e.g. `Skill education-data-source-ccd`, `Read education-data-source-ccd/references/variable-definitions.md`, or for a Bash read `Bash education-data-source-ccd/references/variable-definitions.md` (`Bash education-data-source-ccd/` for the folder; a glob or brace path appears as written after `skills/`, e.g. `Bash education-data-{context,source-ccd}/SKILL.md`, so its first part is then not a skill name); the first word names the kind; null when no retrieval was seen), `retrieval_where` (`main` or `subagent`: every transcript is searched for a `Skill` load or `Read` before any is searched for a Bash read, the main transcript first each time; null when no retrieval was seen), `subagent_transcripts` (the number scanned) and `orchestrator_skill_loaded` (read from the main transcript; null when it is missing, empty or unreadable, as in T3.1); the detail says what matched and where. | **WARN** = the Pre-K answer came with no such retrieval in the main or any subagent transcript: the model may know the fact unaided, so skill retrieval is not proven. Read the main transcript (`transcripts/<sid8>/main.jsonl` in the package) for the specialist dispatch, then the specialist's transcript under `transcripts/<sid8>/subagents/`. Also **WARN** when the main transcript is missing, empty or unreadable after the call, so the resume could not be checked (the detail opens with "The resume is not proven: …", ahead of the retrieval reason; look at where the transcript should be, as for T1.2). **FAIL** = the resume did not continue the T3.1 session: the CLI reported another session id, or the T3.1 transcript has no "Yes, go ahead." user turn (Claude Code may have started a new session; read `transcripts/<sid8>/main.jsonl`). Also **FAIL** when the answer does not say Pre-K, or when the resumed run trips the live-run health gate, which judges the main transcript (no real model answered); a recovered transient error or a `--max-turns` stop turns a PASS into WARN. **SKIP** = T3.1 never reached a real model, so there is nothing to continue. |
---
## Post-run — Shim-Log Verdict for This Run (shim routes)
On a shim route (judged by the **stated** route; a detection mismatch is already T0.1's FAIL), after the live tiers and only when a live tier (1, 2, A or 3) actually ran, the runner scans `scripts/provider_shim/logs/shim.log` again, from the run start (container-local time, the shim's own timestamp format) to the end of the log. This scan carries the shim-log verdict; Tier 0's T0.20 only reports the earlier history, as INFO. It appears in `report.md` as its own **Post-run** section (after every tier section of the per-probe detail), with `tier: "post"` in `report.json` and a Post-run entry on the Passed line. A Tier-0-only, N-only or D-only run has no Post-run section. In `MATRIX.md` the post-run scan has its own row, `T0.20 (post-run)`, below Tier 0's T0.20.
| Probe | PASS means | FAIL / WARN interpretation |
|-------|-----------|----------------------------|
| **T0.20 Shim-log scan (post-run window)** | Counts only, never line text: no failure signal in this run's window. Recovered whitespace retries (`retry_reason=tool_argument_whitespace outcome=success`) are reported as INFO inside the detail. A `shim vX starting` line before the window resets the counters (the scan covers the daemon serving the run); a start **inside** the window, a mid-run restart, does not: the counts accumulate across it, `daemon_restarts_in_window` in the evidence counts it, and the detail adds "Daemon restarts inside the window: N". | **FAIL** = `x-codex-turn-state` lines written during the run (header leakage; shim 1.4.9 stops it; F28), or `insufficient_quota` (an unfunded API key or a spending cap; F22; the counter keeps that name but counts any of the six quota codes the shim classifies: `insufficient_quota`, `usage_not_included`, and since 1.4.11 `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded`, `organization_usage_limit_exceeded`). **WARN** = Claude-slug fast-fails or Claude-named requests (map all aliases, T0.17; F2), unrecovered whitespace trips (`attempts=1 backend_code=excessive_json_whitespace`; F17), `unsupported_value` (for example a `#none` effort; F15) or `context_length_exceeded` (a request over the context ceiling; shim 1.4.10 stops sending the non-streaming in-band case as a 502, and from 1.4.11 the user should see Claude Code's "Prompt is too long" / "Context limit reached" notice (read from client source, not yet seen live); F21, F32); also WARN when the log could not be read. Evidence in the package: `logs/shim.log`, the run window. Read the log around the signal without pasting header lines. |
`POST.CRASH` (FAIL) and `POST.UNAVAILABLE` (SKIP) are the synthetic records for a scan that raised an exception or does not exist in the checkout (see Verdict Semantics).
---
## Route-Conditional Expectations and Known Failure Modes
Each route has a distinct configuration surface. Below: what Tier 0 expects for the route, and the known failure modes (with their documented fixes) that a live-tier FAIL most likely maps to.
### anthropic-subscription (native, no shim)
- **Expected:** No `DAAF_PROVIDER_SHIM`, no OpenRouter base URL. The session model resolves from `ANTHROPIC_MODEL` if set, else a saved `/model` choice, else the soft default `ANTHROPIC_DEFAULT_MODEL` that `settings.json` ships (`opus`, Claude Code's built-in Opus when unmapped). A Claude Enterprise organization default, if set, is not modeled. T0.3 records both start selectors; `ANTHROPIC_MODEL` is normally `<unset>`. `enforce-model-ceiling.sh` actively **ranks** subagent dispatches (haiku < sonnet < opus < fable). T0.2 should report family `claude`, `remap_active=false`.
- **Auth:** Interactive OAuth or `CLAUDE_CODE_OAUTH_TOKEN`. Tier 0 cannot verify interactive OAuth without an LLM call, so a genuine auth problem surfaces at **T1.1** (round-trip fails), and T1.8 confirms the credential in use (`apiKeySource` `none`). `ANTHROPIC_API_KEY` must be empty or unset: a non-empty key makes the install detect as `anthropic-api`, and T0.16 FAILs a stated subscription run because every `-p` probe would bill the key.
- **Known failure mode:** `ANTHROPIC_BASE_URL` unexpectedly set while the route detects as native — T0.3 flags it. Unset it for the native route. A stray `ANTHROPIC_AUTH_TOKEN` outranks the login (T0.3 and T0.16 FAIL).
### anthropic-api (native, Console API key)
- **Expected:** the same endpoint, model resolution and ceiling-hook posture as the subscription route, with a non-empty `ANTHROPIC_API_KEY` as the credential. T1.8 expects a paid `apiKeySource` (`ANTHROPIC_API_KEY`, `/login managed key` or `apiKeyHelper`). A `CLAUDE_CODE_OAUTH_TOKEN` alongside is unused overlap (T0.16 WARN).
- **Spend:** every live probe bills the key, and each call carries `--max-budget-usd` ($2 for Tier 1, $5 per Tier 2 call, $2 for Tier A, $3 per Tier 3 call). T1.7 labels its cost as real spend. A call that hits its cap ends with a non-success result, and the live-run gate FAILs with that reason.
- **Known failure modes:** an unfunded or revoked key fails at T1.1; `ANTHROPIC_AUTH_TOKEN` set outranks the key (T0.16 FAIL).
### openrouter
- **Expected:** `ANTHROPIC_BASE_URL` ends with `openrouter.ai/api`; `ANTHROPIC_AUTH_TOKEN` set (your OpenRouter key, Bearer auth); `ANTHROPIC_API_KEY` **present-and-empty** (`ANTHROPIC_API_KEY=`) so the `X-Api-Key` header does not override Bearer auth. For non-Claude models, map all four tier aliases: the `ANTHROPIC_DEFAULT_OPUS_MODEL`/`SONNET_MODEL` remap keeps subagents pure and makes the ceiling hook **stand down**; `ANTHROPIC_DEFAULT_HAIKU_MODEL` carries Claude Code's housekeeping calls (naming and summarizing the session). Left unmapped, Claude Code has sent these as Claude Haiku requests that the route rejects (July 2026 shim logs); Claude Code's current docs say they run on the main, pricier model. `ANTHROPIC_DEFAULT_FABLE_MODEL` covers the `fable` alias. T0.4 checks every mapped slot's window. `CLAUDE_CODE_SUBAGENT_MODEL` alone does not reroute DAAF agents (a fallback only since Claude Code 2.1.251; `CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1` forces it). `CLAUDE_CODE_MAX_CONTEXT_TOKENS` declares the real window.
- **Known failure modes (documented in `user_reference/01_installation_and_quickstart.md` § "Setup Troubleshooting", anchor `#setup-troubleshooting`):**
- *"model not found" / auth errors* — three causes: wrong base-URL suffix, `ANTHROPIC_API_KEY` unset vs. empty, or a stale login needing `/logout`. T0.3 catches the base-URL and empty-key cases directly.
- *`-pro` GPT slugs* — the July 2026 hard "Prompt is too long" wall (~200k reported tokens) did not reproduce in a 2026-09-29 retest on OpenRouter, which accepted more than 930k reported tokens. A `-pro` "Prompt is too long" at low utilization is therefore not an expected known cause; check the window declaration (T0.4) first. The ~3–4x inflated token accounting persists, so `-pro` runs cost several times more and DAAF's context warnings fire earlier in real content terms (`01_installation_and_quickstart.md` § "GPT (OpenAI) models via OpenRouter (Option C, extended)", anchor `#gpt-openai-models-via-openrouter-option-c-extended`). Prefer a standard flagship slug for cost reasons.
- *Ceiling-hook DENY on a non-Claude session* — if a non-Claude family is configured **without** an OPUS/SONNET remap, a Claude-tier subagent request is denied with remap guidance (`enforce-model-ceiling.sh` rule f). A per-dispatch `model` cannot fix it (the parameter accepts only Claude aliases); set the `ANTHROPIC_DEFAULT_*` remap vars for all four tiers. T0.2's INFO posture is the early warning. `CLAUDE_CODE_SUBAGENT_MODEL` alone does not stand the hook down, and the deny message names the four `ANTHROPIC_DEFAULT_*_MODEL` settings, not that variable. An older installed hook (one without the force flag) does stand down for it; T0.2 says when that hook is the one installed.
- *Context under-reporting* — a GPT/non-`[1m]` slug without a window declaration → T0.4 FAIL.
### chatgpt-subscription and openai-api (shim)
- **Expected:** `DAAF_PROVIDER_SHIM=openai`; `ANTHROPIC_BASE_URL` → `http://127.0.0.1:4141`; `ANTHROPIC_AUTH_TOKEN=daaf-shim-local`; `CLAUDE_CODE_DISABLE_FAST_MODE` exactly `1` (T0.3 FAILs otherwise; `bash /daaf/scripts/provider_shim/gpt_fast.sh off` keeps the requested service-tier policy off). The ChatGPT lane additionally needs `SHIM_BACKEND_MODE=chatgpt` and `CODEX_HOME` (holds `auth.json`); the API lane needs `SHIM_BACKEND_MODE=openai` and `OPENAI_API_KEY` or `SHIM_BACKEND_API_KEY`. T0.8 shim `/health` must report the matching `backend_mode`, a `version` equal to the source on disk, and a valid `gpt_service_tier` block. All four tier aliases must map to GPT models (T0.17), Claude Code must be 2.1.276 or later (T0.15), and TD.7 runs the shim unit suite by default.
- **Daemon state is not per-run overridable.** `SHIM_SANITIZE_TOOLS` and `backend_mode` are fixed at shim startup; `--profiles` overlays reach only the CLI session, so the runner WARNs if `--profiles` is used on a shim route. Verify daemon state read-only via T0.8.
- **Known failure modes:**
- *ChatGPT-lane auth* — `codex_home_present=false`, or T0.9 FAIL (WARN while it is only expiring) → `auth.json` missing, unreadable or expired; run `codex login --device-auth`. Token refresh is JWT-`exp`-based and lazy; a genuinely expired, un-refreshable token surfaces at T1.1.
- *GPT tool-call quirks* — `isolation`-fill hangs and empty `Read.pages` are handled by the shim's `_sanitize_tools` (default on, both shim routes) and, for `isolation`, by `block-remote-isolation.sh` at the hook layer. T2.5 exercises the hook-layer strip. For DAAFBench raw-model runs `SHIM_SANITIZE_TOOLS=0` is required — but that is a benchmark concern, not a deployment-health one.
- *API-lane instant 429* — an unfunded OpenAI API key returns 429 immediately (distinct from a ChatGPT subscription); surfaces at T1.1, and the post-run T0.20 scan FAILs on the `insufficient_quota` log lines. Fund the API account or switch to the chatgpt route (`07_faq_technical.md` § "Q: My GPT session fails instantly with 429 errors on every request (Option F)", anchor `#q-my-gpt-session-fails-instantly-with-429-errors-on-every-request-option-f`).
- *ChatGPT lane is an unofficial dev lane* — explicitly flagged as not OpenAI-sanctioned and liable to break (`01_installation_and_quickstart.md` § "Option F, alternate lane: ChatGPT subscription (Codex backend)", anchor `#option-f-alternate-lane-chatgpt-subscription-codex-backend`); a sudden T1.1 failure here may be upstream, not a local misconfig.
---
## Auto mode
Tiers 1, 2 and 3 use `bypassPermissions`, so they never exercise Auto mode's classifier; Tier A (TA.1) does, with one `--permission-mode auto` call. The notes below explain what TA.1 sees and what an interactive session shows.
- **Working:** allowed actions run with **no permission prompt**. That is the point of Auto mode, so no pop-ups is the expected result. The status bar shows "auto mode on". Explicit ask rules (for example DAAF's `git push`) still prompt.
- **Denied:** Claude Code shows a notification, and the action appears in `/permissions` under *Recently denied*. Three denials in a row, or 20 in total, pause Auto mode and bring prompts back.
- **Classifier failure:** a message saying Auto mode "cannot determine the safety" of an action, with no *Recently denied* entry.
- **On shim routes:**
- Claude Code cannot use Anthropic's server-side review (the `safeguards` pass-through), because the shim's upstream is OpenAI. It falls back to its own classifier requests, which go through the shim to the GPT model. In `shim.log`, a request line containing `stream=n msgs=<N> tools=0` is likely a classifier request, and TA.1 counts successful terminals of that shape. Treat that as a heuristic: other non-streaming, tool-less housekeeping calls share the pattern. The report fingerprint records `CLAUDE_CODE_AUTO_MODE_SERVER`.
- Claude Code announces that fallback with a notice that "this session isn't eligible" for the new classifier billing. On the pinned Claude Code version (2.1.281) the notice is harmless. Set `CLAUDE_CODE_AUTO_MODE_SERVER=0` (see the env template) to skip the server request and the notice.
- After the first reply, `/status` shows "Auto mode server: Disabled". That is expected on these routes.
- **Upgrade gate:** a third party (LiteLLM, October 2026) reports that Anthropic told them Claude Code releases after 2026-10-23 support only server-side Auto mode, which is unavailable behind a gateway that doesn't support it; the shim never can. Anthropic's docs do not say so yet. If it holds, Auto mode stops working on GPT routes after such an update, and `CLAUDE_CODE_AUTO_MODE_SERVER=0` will not restore it. Before bumping the pinned Claude Code version (2.1.281), run Tier A on a shim route and require TA.1 to PASS.
## Where Evidence Lands (for manual follow-up)
When a probe FAILs and the report evidence is not enough, these live locations carry more detail:
| Signal | Location |
|--------|----------|
| Shim health/config | `curl http://127.0.0.1:4141/health` → `backend_mode`, `codex_home_present`, `sanitize_tools`, `reasoning_effort`, `version` |
| Shim structured logs | `scripts/provider_shim/logs/shim.log` (credential-scrubbed) |
| Session/subagent model | `/tmp/claude-model-{session}`, `/tmp/claude-subagent-model-{session}-{id}` |
| Context-window cache | `/tmp/claude-ctx-window-{session}`, `/tmp/claude-or-models-{session}` (OpenRouter models cache) |
| Ceiling-hook stand-down/deny reasons | stderr of `enforce-model-ceiling.sh` and its `permissionDecisionReason` JSON |
| Report artifacts | `scripts/deploy_smoke/reports/{local timestamp}_{stated route}/report.md`, `report.json`, `evidence/` (`shim_health.json`, `env_fingerprint.json`, `t07_statusline_cache/`, `t16_statusline_cache/`, `tier_n/`) |
| Codex config render log | `scripts/provider_shim/logs/codex_config.log` (key names and reasons, never values; T0.18) |
| Diagnostic packages | `scripts/deploy_smoke/packages/daaf-smoke_*.zip` (test container); `scripts/deploy_smoke/inbox/<package>/` and `inbox/MATRIX.md` after `--import` (main container) |
| Tier D failure logs | `scripts/deploy_smoke/reports/{local timestamp}_{stated route}/evidence/tier_d/{probe_id}.log` — complete scrubbed output of any FAILed/timed-out battery; TD.9 keeps each service's own scrubbed output in `evidence/tier_d/td9/{service}.log` |
| Pester NUnit results | `scripts/deploy_smoke/reports/{local timestamp}_{stated route}/evidence/tier_d/testResults.xml` — report-local (never `/daaf/testResults.xml`) |
Reading these `/tmp` caches is the sanctioned pattern (reads allowed, writes blocked). The report's `evidence/` directory already snapshots the shim `/health` JSON and the redacted env fingerprint for offline audit.
---
## The Package: MANIFEST, Scrub Counts, Excluded and Withheld
Every package carries `MANIFEST.json` (`kind: daaf-smoke-package`, `schema_version` 1, `tool_version` from `smoke_package.TOOL_VERSION`, `1.2.0` in this release, bumped whenever a field, a package path or an import rule changes; 1.0.0 packages lack `truncated_reads`, `size` and the newer scrub keys, and packages before 1.2.0 lack `truncated_lines` and dropped whole `| headers:` lines and Codex-header lines in the reports). Read it when something you expected is missing, or before trusting that a file is clean.
**Package name.** `daaf-smoke_<UTC yyyymmddThhmmssZ>_<stated route>_<sha7>.zip`. `<sha7>` is `nosha` when git gave no usable SHA, and a route token that is not a plain route name becomes `unknown-route`. When a package of the same name already exists (two runs in the same UTC second), the packager adds `_2`, `_3`, … before `.zip`; the importer and `MATRIX.md` accept the suffix, and same-route matrix columns show it.
| Field | Meaning |
|-------|---------|
| `package`, `created_utc`, `route`, `detected_route` | The file name, build time, stated route and detected route |
| `run` | `started_utc`, `ended_utc`, the report folder name, `overall` and the verdict counts |
| `versions` | The identity block: DAAF branch, SHA, describe, dirty-file count (`daaf_dirty_files`, tracked and untracked; `daaf_dirty_scope` says so), Claude Code and its pin, Codex, shim and shim source, Python, and `source` (which collector produced the block) |
| `session_ids` | The probe sessions whose transcripts and audit rows were packaged |
| `secret_vocab_source` | `web_retrieval_common` when the shared secret-name vocabulary loaded, `mirror` when the packager used its built-in copy |
| `files[]` | Every packaged file: `path`, `sha256`, `bytes`, `source` (where it came from, with the cut applied, e.g. "run window") and non-zero `scrub_counts` |
| `scrub_totals` | The same counts summed over the package |
| `excluded[]` | Sources deliberately not packaged, each with a `reason` |
| `withheld[]` | Files that were staged and then removed, each with a `reason` |
| `truncated_reads[]` | Logs read from the end because they were larger than the read cap: `source`, `bytes_total`, `bytes_read` |
| `size` | `staged_bytes`, `cap_bytes` (256 MiB) and `cap_reached` |
| `gates` | `post_scrub_secret_scan` (`clean`, or how many files were withheld) and `names_only_invariants` (the `check_workspace_invariants.sh --names-only` result on the staging tree) |
| `manifest_self_scrub_counts` | Present only when scrubbing changed `MANIFEST.json` itself: the replacements made in it (for example `{"email": 1}` for an address in a version string) |
**Scrub counts** (per file and in total) say how many replacements each rule made. Every staged file is scrubbed, including files that are not valid UTF-8: they are decoded with `surrogateescape`, so their undecodable bytes pass through unchanged while text patterns are still replaced.
| Key | What was replaced |
|-----|-------------------|
| `truncated_lines` | `logs/shim.log` only: lines cut at their `\| headers:` dump. The shim writes its backend-error and retry records as `… backend 400 (attempt 1/3) … \| headers: {…}`; the record up to the dump is kept and the dump becomes `\| headers: [REMOVED]`. Cutting runs before the line drops, so a dump holding a Codex header does not cost the record |
| `dropped_lines` | Copied logs and transcripts only: whole lines dropped because they carried `x-codex-turn-state` or `x-codex-credits`. Generated files (`report.md`, `report.json`, the README, `MANIFEST.json`, the settings and quota projections) are never line-dropped: a dropped line would delete a report row or break the JSON. In them a Codex header **with a value** has the value replaced in place (`x-codex-turn-state: [REDACTED:header]`, counted under `secret_header`), and a text that only names the header is kept |
| `env_value` | The live value of a credential-named environment variable (raw or JSON-escaped), replaced with `[REDACTED: value of NAME]` or `<redacted:NAME>` |
| `private_key` | PEM private keys: a whole `-----BEGIN … PRIVATE KEY-----` to `-----END … PRIVATE KEY-----` block, or an orphan BEGIN line with the base64 body after it (a block cut by truncation; raw or JSON-escaped newlines) → `[REDACTED:private-key]` |
| `jwt` | JWT-shaped tokens (`eyJ…`) → `[REDACTED:jwt]` |
| `sk_key` | `sk-`/`rk-` style keys → `[REDACTED:sk-key]` |
| `bearer` | `Bearer <token>` → `[REDACTED:bearer]`, and `Authorization: Basic <credentials>` → `[REDACTED:basic]` |
| `secret_header` | Values of `x-api-key`, `api-key` and `proxy-authorization` → `[REDACTED:header]`; an `Authorization` or `Proxy-Authorization` value with any other scheme or none (a bare token) → `[REDACTED:authorization]`; a Codex turn-state or credits header value in a generated file → `[REDACTED:header]`; and **every** value in a `Cookie` / `Set-Cookie` header, however short (`Cookie: a=1; sid=…` → `Cookie: a=[REDACTED:cookie]; sid=[REDACTED:cookie]`; the names and the `HttpOnly` / `Secure` / `Partitioned` flags are kept, one per redacted value counted). Both the raw form and the JSON-escaped form inside a JSONL string (`\"Authorization\": \"…\"`) match |
| `userinfo` | URL credentials, `scheme://user:password@host` → `scheme://user:[REDACTED:userinfo]@host`, for any host including undotted ones such as `localhost` |
| `url_param` | Secret URL query values, `?key=…` / `&key=…` → `key=[REDACTED:url-param]`, when the key is in the secret vocabulary (hyphens read as underscores) or a signed-URL name (`sig`, `signature`, `code`, `x-amz-signature`, `x-amz-credential`, `x-goog-signature`, `x-goog-credential`, `client_secret`). Non-secret keys such as `api-version` are kept |
| `credits` | The shim's `credits_*` account fields (`credits_has`, `credits_balance`, `credits_unlimited`, …) in `key=value` or `"key": value` form → `credits_<key>=[REDACTED]`, key name kept, in every file, including the per-request quota lines in `logs/shim.log` |
| `email` | Email addresses → `[REDACTED:email]` |
| `secret_assignment` | `NAME=value` or `"name": "value"` where the name is in the secret vocabulary. Kept as diagnostic: session IDs, token counts, `session_model`, the shim's `request_shape` key lists (`keys`, `text_keys`, `reasoning_keys`; key names only), `auth_mode`, `prompt_cache_key`, `cache_key` and the manifest's `secret_vocab_source` (`smoke_package.NON_SECRET_NAMES`, each checked against its writer). `auth_status` and `key_source` stay redacted: no writer was found to check them against. A value may contain balanced `[…]` groups, so a model ID such as `gpt-6.1-sol[1m]#high` is read as one value, never cut at `]` |
A high `email` or `secret_assignment` count in a transcript usually means the model quoted configuration or a user address; it is not by itself a leak, since the values were replaced. Placeholders such as `<redacted:set>` in the fingerprint and settings projection come from the report's own name-based redaction.
**The post-scrub gate** re-scans every staged file with the same shapes the scrubber applies (PEM BEGIN lines, JWTs, `sk-`/`rk-` keys, Bearer, Basic, the named headers, Authorization, URL userinfo, secret URL parameters, `credits_*` values, emails, secret-vocabulary assignments), plus Cookie values, any leftover `x-codex-turn-state` / `x-codex-credits` text in a copied log or transcript (in a generated file, only such a header with an unredacted value), and any live credential env value. A generated JSON file that scrubbing changed must still parse; if it does not, it is withheld as `withheld: generated JSON no longer parses after scrubbing (…)`. None of those patterns matches the scrubber's own `[REDACTED…]` markers or route_detection's `<redacted…>` markers, so a scrubbed file does not trip the gate on its own replacements.
**Size and read caps.** A copied file over 64 MiB (`MAX_SOURCE_FILE_BYTES`) is excluded as too large. The three growing logs are not excluded but read **from the end**, at most 64 MiB each, before they are cut: `shim.log` (then the run window), `codex_config.log` (then its last 200 lines) and `activity.log` (then the run window, at most its last 2000 lines). A log larger than the cap is listed in `truncated_reads`. The staged package as a whole is capped at 256 MiB (`MAX_PACKAGE_TOTAL_BYTES`): files past the cap are excluded with the reason "package size cap reached", the run prints a WARN, and `size.cap_reached` is true. The generated README is exempt from the cap.
**Excluded** sources are policy, recorded so the reader knows they were deliberate: `$CODEX_HOME/auth.json`, `~/.claude/.credentials.json` and `.claude.json`, the shim reasoning cache, `scripts/provider_shim/logs/diagnostics/`, the bulk session archive (only probe sessions are packaged), rotated `shim.log.[1-3]`, the `shim.stream.*` FIFOs, any file whose name marks credential material (any name starting with `.env`, `*.pem`, `*.key`, `credentials*`, `secrets`, `environment_settings*`, SSH keys), non-regular files (symlinks, FIFOs, sockets), files over 64 MiB, files past the package size cap, and sources that were simply not present (for example no transcript found, or no shim log off the shim routes).
**Withheld** files were staged and then removed: `withheld: secret-shaped content (<kinds>)` when the post-scrub gate still found a secret shape after scrubbing (the run prints a WARN naming the file), a name Windows cannot create or a case-only clash, a duplicate package path, or a name that failed `--names-only`. If the names check still fails after those removals, no package is written at all. A withheld transcript is the usual case: open the original on the test container if the evidence matters, and never copy it out unscrubbed.
**README and report annotations.** `README_FOR_MAIN_SESSION.md` is generated **after** both gates. Its **"Not in this package"** section lists the withheld files with their reasons (or "none"), the sources not found when the package was built, the logs read from the end only (bytes read of bytes total), and whether the size cap was reached; it then points to `MANIFEST.json` for every exclusion. The README is scrubbed and gate-checked like any file; if a listed path or reason trips a pattern, it is regenerated as static text, which is scrubbed and gate-checked again, and if that also fails the README is withheld (reason "full and static forms", with a WARN). `MANIFEST.json` is written last, after every other file is final, and is itself scrubbed and gate-checked; it never hashes itself, so scrubbing it cannot break a file hash. If scrubbing would change its file list, leave it unparseable or unstable, or the gate still finds a secret shape in it, no package is written (the run prints the reason as a WARN). The packaged `report.md` is annotated at the same point: in its Problems table, an evidence path the zip does not hold is marked `(withheld)` or `(not in package)`.
**Importer checks** (`--import`): the file must be a regular `.zip` with a safe name; members must be regular files with safe relative names (no absolute paths, `..`, backslashes, NUL or Windows-incompatible names), each listed in `MANIFEST.json` and matching its sha256 and size; limits are 512 MiB per member and 1 GiB per package. Any failure rejects the whole package and keeps nothing. A corrupt or unreadable package is rejected the same way (`corrupt zip (…)` or `extraction failed (<error type>: …)`), and the importer goes on to the next package: one bad package never blocks the others.
---
## Failure Catalogue → Probe Map
The GPT-shim failure catalogue (F1-F32: F1-F31 are the failure modes found in a 2026-10-13 audit of the shim, F32 the context-overflow handling seen in a live context-ceiling probe run the same day) and the probes that catch each row. TD.7 tests the shim source on disk; the Tier 0 shim probes look at the running daemon and its log. T0.20 below means the **post-run** scan of this run's window, which carries the verdict; the Tier 0 scan of earlier history is INFO.
| Failure | Caught by |
|---------|-----------|
| F1 Session opens on a Claude model on a GPT route | T0.14, T0.12 |
| F2 Claude-named requests rejected | T0.17 (configuration), T0.20 (fast-fails and Claude-named requests in the log) |
| F3 Housekeeping calls sent as Claude Haiku | T0.17 |
| F4 "Prompt is too long" at low use | T0.4 |
| F5 Inflated `count_tokens` estimate | TD.7 |
| F6 Stale daemon after an update | T0.8 (version against source) |
| F7 Codex config not rendered or not logged | T0.18 |
| F8 ChatGPT login expired or absent | T0.9 |
| F9 Auth refresh race | T0.9 (end state), TD.7 |
| F10 New model not registered in the context hooks | TD.0 (lockstep test), T0.4 (partial) |
| F11 Statusline misreads an effort suffix | TD.1 (statusline bats) |
| F12 Auto-mode "isn't eligible" notice | TA.1 (classifier path); fingerprint records `CLAUDE_CODE_AUTO_MODE_SERVER` |
| F13 Auto mode lost after a Claude Code bump | TA.1 |
| F14 Empty reply from a non-streaming call | TD.7 |
| F15 `#none` effort rejected | TD.7, T0.20 (`unsupported_value`) |
| F16 Runaway whitespace in tool arguments | T0.20 (unrecovered trips), TD.7 |
| F17 Whitespace trip with no retry | TD.7, T0.20 |
| F18 Tool run with empty arguments | TD.7 |
| F19 Strict validators reject the real wire format | T1.1, T2.2 |
| F20 `role:system` message rejected | T1.1 |
| F21 Context overflow retried as a 502 (shim 1.2.10 fixed the case rejected before the reply; the in-band non-streaming case is F32; since 1.4.11 the message reads "Prompt is too long (backend context length exceeded)") | TD.7, T0.20 (`context_length_exceeded`) |
| F22 Instant 429 on an unfunded key (since shim 1.4.11 a non-streaming reply carries `x-should-retry: false`, so Claude Code stops retrying it, except with `CLAUDE_CODE_RETRY_WATCHDOG` set, which still retries a 429; a streamed reply cannot carry the header, but a streamed main-loop request that fails before any content is re-sent once without streaming and that re-send carries it, so the error costs one streamed attempt plus one non-streamed re-send; both read from the 2.1.281 client source, not observed live. OpenAI's newer billing codes such as `credit_balance_exhausted` are classified the same way) | T0.20 (`insufficient_quota`), T1.1 |
| F23 `/model` switch to GPT fails | TD.7 |
| F24 Stale reasoning blob after a restart | TD.7 |
| F25 WebSearch rejected or failing | T2.3 (WARN on a fallback-only pass on shim routes) |
| F26 WebSearch schema strictness | TD.7 |
| F27 Attached PDF breaks later turns | TD.7 |
| F28 Session tokens written to `shim.log` | TD.7, T0.20 (`x-codex-turn-state` lines) |
| F29 Prompt cache near zero | T0.19, TD.7 |
| F30 Pasted gateway URL fails | TD.7 |
| F31 Claude Code inside the broken 2.1.265-2.1.275 window | T0.15 (2.1.276 floor), T0.5 |
| F32 Non-streaming in-band context overflow sent as a 502 (fixed in shim 1.4.10, confirmed live 2026-10-14 with a direct probe: HTTP 400, attempts=1; 1.4.11 makes the message one Claude Code recognizes as a context overflow) | TD.7, T0.20 (`context_length_exceeded`) |
---
## Unconfirmed Until First Live Run
These thresholds and match strings come from design inference, documentation summaries or a single observation. Until a live run on the named route confirms them, read a FAIL or WARN that rests on one of them as "check the transcript", not as proof.
| Probe | What is unconfirmed |
|-------|---------------------|
| T1.8 | The `apiKeySource` values for paid credentials (`ANTHROPIC_API_KEY`, `/login managed key`, `apiKeyHelper`) come from documentation summaries; only `none` has been observed (subscription route). |
| T1.9 | The WARN case (the skill loaded but no reminder attachment was recorded) has not been seen on a live run. The INFO result is confirmed (below). |
| T2.7 | The deny shapes. `BLOCKED by bash-safety hook` is confirmed in the hook source. The Bash permission phrase ("Permission to use Bash with command … has been denied") was observed on 2026-09-29 in a stored subagent transcript for a different deny rule (`Bash(printenv*)`). The Read phrase ("Permission to read … has been denied") comes from the 2.1.261 CLI source only; "File is in a directory that is denied by your permission settings." was found by a reviewer in the installed 2.1.281 binary (Read's input validation), not in a transcript. None has been seen in a T2.7 run on 2.1.281; a deny in another wording shows as WARN with the text quoted. pip's own output strings (the "pip ran" FAIL) are inferred from pip's messages. |
| Tier 1, 2, A, 3 on `anthropic-api` | The `--max-budget-usd` caps ($2, $5 per call, $2, $3 per call) were chosen with headroom, not measured; a cap that is too tight shows as a gate FAIL. |
| T0.19 | The 0.3 ratio and the 20-request minimum (a healthy ChatGPT-lane session measured about 0.92). |
| T0.20 (Tier 0 and post-run) | The log signal strings (`claude-slug fast-fail`, `attempts=1 backend_code=excessive_json_whitespace`, `retry_reason=tool_argument_whitespace outcome=success`, and the rest) follow the shim's log format and earlier audit recipes; they have not been exercised by a smoke run. The post-run window starts at the run start in container-local time and assumes the shim's timestamps use the same clock. |
| TA.1 | The classifier signature on shim routes (`event=terminal`, `outcome=success`, ` stream=n `, ` tools=0 `). Nothing in the shim's terminal record marks the Auto-mode classifier, and Claude Code's other non-streaming, tool-less calls (session titles, resume summaries) match too, so a PASS can occur without a classifier call. No tighter signature exists in the log format (checked 2026-10-13); revisit once a live TA.1 run is read. |
| T3.1 | Neither WARN case (`daaf-orchestrator` not loaded in a real session; the load not checked because the main transcript is missing, empty or unreadable) has been seen on a live run, and the other four routes are unchecked. On `anthropic-subscription` the PASS signal, a non-error `daaf-orchestrator` `Skill` load in the T3.1 turn, is confirmed in three runs, one of them graded by the orchestrator-load rule (below). |
| T3.2 | The other four routes: that the orchestrator hands the lookup to a specialist there too, that the Pre-K pattern matches their answers, and that `--resume <session>` keeps the session id and appends the T3.2 turn to the T3.1 transcript there as well (the resume check depends on it). On `anthropic-subscription` the resume shape, the Pre-K answer, a PASS under the main-plus-subagent scan and a PASS graded by the resume check are confirmed (below). Not yet seen on a live run: either FAIL branch of the resume check (another session id, a missing T3.2 turn), the unchecked-resume WARN, a graded report whose retrieval signal only the `education-data-*` prefix rule counts (a skill other than `education-data-context` and `education-data-source-ccd`, or a `Read` of a reference page), and a graded session whose only retrieval is a Bash read (the Bash reads in sessions `d99a40c0` and `6be2cfa9`, below, are outranked by a specialist's `Skill` load). |
| Tier 3 cost (identity block) | Whether a resumed session's reported `total_cost_usd` is cumulative (T3.1 plus T3.2). The runner assumes it may be and labels the total an upper bound (`total_cost_is_upper_bound`); if it is not cumulative, the label is merely cautious. |
| TN.3 | The verdict words `CONFIDENT` and `WEAK`; only `EXISTS` has been seen (below). |
| T0.21 | The 5 GB free-space warning level. |
| Package | Log windows assume container-local timestamps in `shim.log` and `activity.log`; the shim window starts at the later of run start minus 5 s and the last daemon start. The URL-parameter, userinfo, scheme-less Authorization and PEM scrub patterns are unit-tested on synthetic text, not yet seen on a real run's files (the `credits` and `email` patterns are, below). |
| Preflight gate | A live block: no run has yet tripped the gate, so `T0.ABORT`'s wording, the "Tier 0 (gate subset)" section and the `--allow-preflight-fail` override are confirmed by the self-tests only. The same holds for the profile gate, which no live `--profiles` run has exercised yet. |
| SIGTERM / SIGHUP during a run | Confirmed by the self-tests (a runner process holding a TD.9 stub service, signalled mid-poll: the service is stopped, `RUN.CRASH` is written, the exit code is 143 or 129; and the same runner on a pseudo-terminal whose terminal end is closed, where the package is still built and the exit code is 129), not yet on a maintainer's live run. |
| T0.0, T0.15, T0.22, T0.24, TD.9 on the standard image and on arm64 | All five have run only on the developer image, on amd64 (one run, 2026-11-01, below). The `standard image` flavor, the INFO downgrade for missing dev tools, and TD.9's 45 s answer window on a slower arm64 runner (it answered in under 1 s on amd64) are unobserved. |
**Confirmed on a live run.** Each item below, with the run that showed it. The report folders are maintainer-local (`scripts/deploy_smoke/reports/` and `packages/` are gitignored, so they do not ship):
| Item | Observed | Where |
|------|----------|-------|
| T1.9 INFO on a healthy install | `[INFO] T1.9 — Orchestrator bootstrap`, evidence `{"reminder_injected": true, "orchestrator_skill_loaded": false}`: the reminder matched, the skill was not loaded | `chatgpt-subscription`, 2026-10-30; report folders `20261030_190449_chatgpt-subscription` and `20261030_190647_chatgpt-subscription` |
| TN.3 `EXISTS` | `scholar_search verify resolved 10.1038/nature14539 (EXISTS).`, evidence `{"exit": 0, "verdict": "EXISTS"}` | the same 2026-10-30 runs, also `20261030_190328_chatgpt-subscription` |
| TN.4 at the pinned revision | `Education-data mirror reachable (HTTP 302).` for `saipe/districts_saipe.parquet` at the pinned revision (a redirect to the file store, below 400) | the same three 2026-10-30 runs |
| `credits` and `email` scrub patterns on real files | Package `daaf-smoke_20261030T190647Z_chatgpt-subscription_72c8700.zip` (run folder `20261030_190647_chatgpt-subscription`): `MANIFEST.json` `scrub_totals` `"credits": 9` (6 in `logs/shim.log`, 3 in TN.3's saved scholar_search files) and `"email": 2` (in the probe transcript), with `"post_scrub_secret_scan": "clean"` | 2026-10-30 |
| T0.0, T0.15, T0.22, T0.24, TD.9 on the developer image | `[INFO] T0.0 — Image flavor`: "Image flavor: developer image (bats, pwsh, shellcheck and gh present); Tier D runs every battery."; `[PASS] T0.24 — Settings invariants`; `[PASS] TD.9 — Local browser services`: "marimo: HTTP 303 in 0.5s; code-server: HTTP 302 in 0.7s; launchers pass bash -n and invoke the tested binaries." The same run's `[FAIL] TD.1` ("bats tests/bash timed out after 600s.") is the TD.1 known issue | the developer container, which detects as `anthropic-subscription`; that run stated `--route chatgpt-subscription`, so T0.1 FAILed as a mismatch. Tiers 0 and D, 2026-11-01; report folder `20261101_164142_chatgpt-subscription` (the folder carries the stated route). T0.0 and T0.24 gave the same results with the matching route in `20261101_183323_anthropic-subscription` (Tier 0 only) |
| T3.1 `daaf-orchestrator` load in the T3.1 turn | In both runs the T3.1 turn's main transcript loaded `daaf-orchestrator` with the `Skill` tool before the reply, and the reply named Data Lookup and ended `**Sound good?**`. Session `17375d34`: the main transcript's tool calls were `Skill` `daaf-orchestrator`, `Read`, `Agent` (`source-researcher`), `Bash`, `Read`, the `Skill` call coming before the "Yes, go ahead." turn. Session `d99a40c0`: the `Skill` call came before the reply, and T3.2's evidence records `"orchestrator_skill_loaded": true`. The `d99a40c0` reply mentions Pre-K only in a scope example ("such as how Pre-K coverage changes across years or states, I'll ask before widening the scope"), so its `prek_already_answered` is true although the reply does not answer: one reason gate behaviour is recorded, not graded. That reply is a regression case in the harness self-tests (PASS under the orchestrator-load rule) | `anthropic-subscription`; 2026-11-01, report folder `20261101_230944_anthropic-subscription`; 2026-11-02, report folder `20261102_032432_anthropic-subscription` |
| T3.2 Pre-K answer, `--resume`, and retrieval in the main and a specialist transcript | `claude -p --resume 17375d34 'Yes, go ahead.'` continued the T3.1 session in print mode, and the answer said `grade = -1` means Pre-Kindergarten. The main transcript's tool calls: `Skill` `daaf-orchestrator`, `Read` (`daaf-orchestrator/references/data-lookup-mode.md`), `Agent` (`source-researcher`), `Bash`, `Read` (`education-data-source-ccd/references/variable-definitions.md`, non-error). The specialist's transcript under `~/.claude/projects/-daaf/<session>/subagents/` held a `Skill` load of `education-data-source-ccd` and a `Read` of `/daaf/.claude/skills/education-data-context/SKILL.md`, both with non-error results. Under the `education-data-*` rule the main transcript's `Read` of the CCD reference page counts, so this session shows retrieval in both places. That run's own report was graded by the rule of that run (a `Skill` load of `education-data-context` in the main transcript) and shows `[WARN] T3.2`, evidence `{"prek_answer": true, "context_skill_loaded": false, "orchestrator_skill_loaded": true}`. In sessions `d99a40c0` and `6be2cfa9` (below) each main transcript loads only `daaf-orchestrator` and reads only `data-lookup-mode.md`, so a `Skill` load or `Read` of an education-data skill appears only in the specialist's transcript, which is why T3.2 also scans subagent transcripts. Each main transcript also holds a counted Bash read: a `grep` naming `education-data-source-ccd/references/variable-definitions.md` (with two more CCD skill files in `d99a40c0`) that returned matching lines, among them the line saying `grade = -1` means Pre-K. The specialist's `Skill` load of `education-data-source-ccd` outranks it, so it stays the recorded signal (the T3.2 evaluator run over the archived transcripts on 2026-11-02 gives `Bash education-data-source-ccd/references/variable-definitions.md` in `main` with the subagent transcripts left out, and `Skill education-data-source-ccd` in `subagent` with them) | `anthropic-subscription`, 2026-11-01; report folder `20261101_230944_anthropic-subscription` |
| T3.2 PASS under the main-plus-subagent scan, and the resume shape | `[PASS] T3.2 — Golden path (skill retrieval)` (the probe's name in that run's report), detail "The answer says grade -1 is Pre-K, with skill retrieval in a specialist (subagent) transcript: Skill education-data-source-ccd (daaf-orchestrator loaded too).", evidence `"retrieval_signal": "Skill education-data-source-ccd", "retrieval_where": "subagent", "subagent_transcripts": 1`. The resume shape the resume check relies on: after `claude -p --resume d99a40c0 'Yes, go ahead.'`, the session's single transcript file held the T3.2 prompt as a user record (`"content":"Yes, go ahead."`) whose `parentUuid` is the T3.1 turn's last record, and all 61 records in the file carry `sessionId` `d99a40c0-0875-4c55-b1ef-c58135df9c20`; session `17375d34` shows the same single-file shape | `anthropic-subscription`, 2026-11-02, Claude Code 2.1.281; report folder `20261102_032432_anthropic-subscription` |
| T3.1 PASS under the orchestrator-load rule, and T3.2 PASS with the resume check | `[PASS] T3.1 — Orchestrator engaged`, detail "daaf-orchestrator loaded in the main transcript. Recorded, not graded: mode named: Data Lookup; asks confirmation: yes; Pre-K mentioned: no.", evidence `{"orchestrator_skill_loaded": true, "mode_named": "Data Lookup", "asks_confirmation": true, "prek_already_answered": false}`. `[PASS] T3.2 — Golden path (resume + skill retrieval)`, detail "The answer says grade -1 is Pre-K, with skill retrieval in a specialist (subagent) transcript: Skill education-data-source-ccd (daaf-orchestrator loaded too).", evidence `"resumed_same_session": true, "retrieval_signal": "Skill education-data-source-ccd", "retrieval_where": "subagent", "subagent_transcripts": 1`. Session `6be2cfa9`: the main transcript's only `Skill` load is `daaf-orchestrator` and its only `Read` is `data-lookup-mode.md` | `anthropic-subscription`, 2026-11-02, Claude Code 2.1.281; report folder `20261102_040645_anthropic-subscription` |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |