Skip to content

air-gapped/skills / .claude/skills/claude-code-ci-harness

claude-code-ci-harness

Unverified

Run Claude Code headlessly as an isolated CI harness against any Anthropic Messages-compatible endpoint (local engine, gateway or hosted API) or a Claude subscription token. Covers credential isolation, stream-json transcripts, tool-call assertions, multi-turn scenarios, and endpoint/model compatibility diagnosis.

Claude Code CI Harness

Never invoke claude directly for a test. Always use scripts/isolated-run.sh. It isolates the run from the caller's profile.

Run

${CLAUDE_SKILL_DIR}/scripts/isolated-run.sh MODE_FLAGS \
  --workdir DIR --transcript OUT.ndjson --prompt 'PROMPT'
ModeMODE_FLAGS
Endpoint, no auth--base-url URL --model SERVED_NAME
Endpoint, bearer token--base-url URL --model SERVED_NAME --token-var VAR
Endpoint, X-Api-Key header--base-url URL --model SERVED_NAME --token-var VAR --x-api-key
Claude subscription--model CLAUDE_ALIAS_OR_ID (no --base-url; token var defaults to CLAUDE_CODE_OAUTH_TOKEN)
FlagRule
--base-urlEndpoint root without /v1, local or remote.
--modelThe exact name the endpoint serves; a Claude alias or ID in subscription mode.
--token-varName of a variable in the secrets file.
--x-api-keyOnly for endpoints that require the token in an X-Api-Key header.
--toolsBuilt-in tool allowlist. Default: Read,Glob.
--timeoutWall-clock limit in seconds. Default: 180. 0 disables it.

Secrets

The secrets file is ~/.config/claude-code-ci-harness/targets.env. Each line is KEY=value. The file must be mode 0600, owned by the user, not a symlink, and outside Git. The runner rejects any other file.

  • CI job: write the file from the CI secret store with umask 077 to a path outside the checkout (for example $RUNNER_TEMP/targets.env). Pass it with CLAUDE_CODE_CI_SECRETS_FILE=PATH, which run-matrix.sh also honours, or --secrets-file PATH. A file inside the checkout is rejected as inside Git.
  • Subscription token: the user runs claude setup-token in a separate terminal and adds CLAUDE_CODE_OAUTH_TOKEN=<token> to the file. Never run setup-token inside an agent session, because the token would land in the transcript.
  • Never copy ~/.claude/.credentials.json into a run. Its refresh token rotates, and a rotated copy logs out the user.
  • Never put a token in a flag, a transcript or the skill directory.

Do not change in the runner

KeepReason
--safe-modeDisables CLAUDE.md, skills, plugins and hooks.
--permission-mode dontAskDenies every tool not in --allowed-tools. The -p default mode changes between releases.
--strict-mcp-config with an empty configBlocks the caller's MCP servers.
Empty ANTHROPIC_API_KEYStops an inherited key from overriding the intended token.
No --bareIt disables OAuth and has dropped Glob from --tools.

Test

  1. Build the workdir: ${CLAUDE_SKILL_DIR}/scripts/make-fixtures.sh DIR.
  2. Run the smoke scenario. Then escalate through references/scenario-bank.md, and stop at the first failure confirmed per step 6.
  3. Assert on every transcript: ${CLAUDE_SKILL_DIR}/scripts/assert-events.sh OUT.ndjson [--expect-tools Glob,Read] [--expect-model NAME] [--min-turns N] [--max-turns N] [--min-parallel N]. It checks result success without is_error, tool sequence, model, turn bounds and tool calls per assistant message. Exit status is not evidence.
  4. For more than one endpoint: ${CLAUDE_SKILL_DIR}/scripts/run-matrix.sh --prompt-file PROMPT.txt --workdir DIR --output-dir OUT --run 'label|base-url|model[|token-var]' [--run ...]. An empty base-url means subscription mode. A run passes only if its transcript passes assert-events.sh; add scenario assertions per step 3.
  5. For an open-weights model, scan every transcript for leaked special tokens: ${CLAUDE_SKILL_DIR}/scripts/scan-token-leakage.sh OUT.ndjson MODEL_DIR. MODEL_DIR holds the served model's tokenizer_config.json, tokenizer.json or *.jinja; a file with one token per line also works. Exit 1 means a leak.
  6. A single failing run is not a finding. Re-run that scenario until it has 3 transcripts and read them as a set: failing in 2 or 3 of 3 is an endpoint/model property; failing in 1 of 3 is noise unless its transcript shows the mechanism (name the event where it diverges).
  7. A run that delivers nothing (no result event, or an empty final answer) outranks every other result. A runner or setting change after which any run delivers nothing is unsafe: revert it, even if other runs improved.
  8. When an endpoint fails, record it as an endpoint/model compatibility finding. Never change the scenario to make it pass.
  9. Afterwards, ${CLAUDE_SKILL_DIR}/scripts/cleanup.sh --run-dirs removes run directories that a killed run left in /tmp.

Done when every endpoint has a result row: claude --version, base URL, model, and for every scenario either pass or the first failing assertion or error event, with the pass count when it was re-run (2/3).

References

  • references/scenario-bank.md: scenarios and pass criteria
  • references/known-gaps.md: symptom → cause → check
  • references/isolation.md: what the runner isolates, for anyone editing it
  • references/sources.md: verification stamps (freshen mode only)
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 319kUpdated Claude CodeCodexVerifiedInstall