air-gapped/skills / .claude/skills/claude-code-ci-harness
claude-code-ci-harness
UnverifiedRun Claude Code headlessly as an isolated CI harness against any Anthropic Messages-compatible endpoint (local engine, gateway or hosted API) or a Claude subscription token. Covers credential isolation, stream-json transcripts, tool-call assertions, multi-turn scenarios, and endpoint/model compatibility diagnosis.
Claude Code CI Harness
Never invoke claude directly for a test. Always use
scripts/isolated-run.sh. It isolates the run from the caller's profile.
Run
${CLAUDE_SKILL_DIR}/scripts/isolated-run.sh MODE_FLAGS \
--workdir DIR --transcript OUT.ndjson --prompt 'PROMPT'
| Mode | MODE_FLAGS |
|---|---|
| Endpoint, no auth | --base-url URL --model SERVED_NAME |
| Endpoint, bearer token | --base-url URL --model SERVED_NAME --token-var VAR |
Endpoint, X-Api-Key header | --base-url URL --model SERVED_NAME --token-var VAR --x-api-key |
| Claude subscription | --model CLAUDE_ALIAS_OR_ID (no --base-url; token var defaults to CLAUDE_CODE_OAUTH_TOKEN) |
| Flag | Rule |
|---|---|
--base-url | Endpoint root without /v1, local or remote. |
--model | The exact name the endpoint serves; a Claude alias or ID in subscription mode. |
--token-var | Name of a variable in the secrets file. |
--x-api-key | Only for endpoints that require the token in an X-Api-Key header. |
--tools | Built-in tool allowlist. Default: Read,Glob. |
--timeout | Wall-clock limit in seconds. Default: 180. 0 disables it. |
Secrets
The secrets file is ~/.config/claude-code-ci-harness/targets.env. Each line is KEY=value. The file must be mode 0600, owned by the user, not a symlink, and outside Git. The runner rejects any other file.
- CI job: write the file from the CI secret store with
umask 077to a path outside the checkout (for example$RUNNER_TEMP/targets.env). Pass it withCLAUDE_CODE_CI_SECRETS_FILE=PATH, whichrun-matrix.shalso honours, or--secrets-file PATH. A file inside the checkout is rejected as inside Git. - Subscription token: the user runs
claude setup-tokenin a separate terminal and addsCLAUDE_CODE_OAUTH_TOKEN=<token>to the file. Never runsetup-tokeninside an agent session, because the token would land in the transcript. - Never copy
~/.claude/.credentials.jsoninto a run. Its refresh token rotates, and a rotated copy logs out the user. - Never put a token in a flag, a transcript or the skill directory.
Do not change in the runner
| Keep | Reason |
|---|---|
--safe-mode | Disables CLAUDE.md, skills, plugins and hooks. |
--permission-mode dontAsk | Denies every tool not in --allowed-tools. The -p default mode changes between releases. |
--strict-mcp-config with an empty config | Blocks the caller's MCP servers. |
Empty ANTHROPIC_API_KEY | Stops an inherited key from overriding the intended token. |
No --bare | It disables OAuth and has dropped Glob from --tools. |
Test
- Build the workdir:
${CLAUDE_SKILL_DIR}/scripts/make-fixtures.sh DIR. - Run the smoke scenario. Then escalate through
references/scenario-bank.md, and stop at the first failure confirmed per step 6. - Assert on every transcript:
${CLAUDE_SKILL_DIR}/scripts/assert-events.sh OUT.ndjson [--expect-tools Glob,Read] [--expect-model NAME] [--min-turns N] [--max-turns N] [--min-parallel N]. It checks result success withoutis_error, tool sequence, model, turn bounds and tool calls per assistant message. Exit status is not evidence. - For more than one endpoint:
${CLAUDE_SKILL_DIR}/scripts/run-matrix.sh --prompt-file PROMPT.txt --workdir DIR --output-dir OUT --run 'label|base-url|model[|token-var]' [--run ...]. An empty base-url means subscription mode. A run passes only if its transcript passesassert-events.sh; add scenario assertions per step 3. - For an open-weights model, scan every transcript for leaked special tokens:
${CLAUDE_SKILL_DIR}/scripts/scan-token-leakage.sh OUT.ndjson MODEL_DIR.MODEL_DIRholds the served model'stokenizer_config.json,tokenizer.jsonor*.jinja; a file with one token per line also works. Exit 1 means a leak. - A single failing run is not a finding. Re-run that scenario until it has 3 transcripts and read them as a set: failing in 2 or 3 of 3 is an endpoint/model property; failing in 1 of 3 is noise unless its transcript shows the mechanism (name the event where it diverges).
- A run that delivers nothing (no
resultevent, or an empty final answer) outranks every other result. A runner or setting change after which any run delivers nothing is unsafe: revert it, even if other runs improved. - When an endpoint fails, record it as an endpoint/model compatibility finding. Never change the scenario to make it pass.
- Afterwards,
${CLAUDE_SKILL_DIR}/scripts/cleanup.sh --run-dirsremoves run directories that a killed run left in/tmp.
Done when every endpoint has a result row: claude --version, base URL, model, and for every scenario either pass or the first failing assertion or error event, with the pass count when it was re-run (2/3).
References
references/scenario-bank.md: scenarios and pass criteriareferences/known-gaps.md: symptom → cause → checkreferences/isolation.md: what the runner isolates, for anyone editing itreferences/sources.md: verification stamps (freshen mode only)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |