Pixel-CLI/pixel / .agents/skills/ultima-arena
ultima-arena
VerifiedInstall Pixel from a repository, start Claude Code, Codex, pi, agy and Devin in one Herdr grid in random order, send each the same neutral prompt, and judge per CLI from evidence only whether the prompt-submit brief and a Pixel metrics line reached it. Use for /ultima-arena, 'ultima arena', 'test brief and metrics across all CLIs', or after a change to a hook, relay or install.
ultima-arena: brief and metrics, five CLIs, evidence only
ultima_arena.py (next to this file) answers two questions for each CLI,
from files and screens, never from what the model says:
| Check | PASS when |
|---|---|
brief | logged: <repo>/.pixel/brief-feedback.jsonl holds a prompt line and a part line with bytes > 0 of that CLI's provider, both after the prompt; and delivered: the pane shows brief text or the prompt-submit notice, Devin shows its inspected hook-completed line, Pi persists its pixel-brief custom message, or agy stores the injected brief (below). |
metrics | the pane, Pi's paired bash tool result, or agy's injected store step contains 🟩 pixel search-content … #<id>, and <repo>/.pixel/actions.jsonl holds a search-content record after the prompt with that invocation ID. Pi also checks the exact command and repository. |
agy draws no hook output. agy 1.4.0 shows nothing a hook returns: of every response
shape tried, only an injected userMessage step appears, and it appears as a fake user turn.
verbosity: high adds only thinking lines. So for agy the arena reads, beside the screen,
what agy stored: ~/.gemini/antigravity-cli/conversations/<conversationId>.db, where the
conversation id is the session of the turn's prompt line (or, when no line names one, every
file written since the prompt). Only steps of type 90 count: each one is an injected
ephemeralMessage. The tool's own output is type 132, so its box never passes for the relay.
[PIXEL:BRIEF] there is the brief (delivery_evidence: "agy store: [PIXEL:BRIEF]"), and a 🟩
box there is the relay. The receipt lists the conversations read (agy_store), and
agy.injected.txt holds the steps. This proves the brief and the relay reached agy's model;
agy's user sees neither.
NO-CALL (metrics) means the model answered without running the Pixel
command: there was nothing to relay, so it is neither a pass nor a relay
failure, and the run still exits 1. The prompt asks for the command before the
question for that reason.
The prompt requires the exact Pixel shell command even when the brief or
prior context already contains the answer; it says to report NOT RUN if the
command cannot run. This prevents a correct answer from hiding a missing tool
call. It never mentions metrics (a hard guard refuses metric, 🟩, ❀),
and the answer is never scored: a model asked about metrics says yes.
Run the repo workflow from a Herdr operator pane. just is optional; the
script works directly too:
just ultima-arena # fixed seed 42, install, then run the five-CLI pane grid
just ultima-arena --skip-install # same fixed seed, use the Pixel already on PATH
just ultima-arena --seed 7 # choose and record another fixed start order
just ultima-arena abort # close only panes created by the last run
just ultima-arena --self-test # offline contract, no Herdr, no model
scripts/dev/ultima-arena.sh --seed 42 # same runner without just
The default seed 42 fixes agent start order and is recorded in the receipt;
model outputs remain live. The runner submits the prompt to every ready CLI
before monitoring any turn. It then monitors all panes and interrupts each CLI
after its submitted brief and matching metrics relay are evidenced.
scripts/dev/ultima-arena.sh resolves this
checkout and forwards all harness options. For a live run, it requires
HERDR_PANE_ID from the operator pane so the grid is created beside that pane.
Flags: --repo PATH (default: git toplevel of the cwd; may be another checkout
than the script's), --skip-install, --seed N (default 42, printed
and stored), --clis claude,codex,pi,agy,devin, --timeout SECS per CLI
(default 300), --out DIR (default <repo>/eval/arena-results/ultima-<UTC>/,
gitignored). Exit 0 when every CLI passes both checks, 1 otherwise, 2 for a
usage or environment problem (no Herdr pane, PIXEL_BRIEF_LOG off, install failed).
What a run does
- Overlap warning and install: before any install or pane changes,
herdr agent listis checked for working agents in the same repo/worktree family as<repo>; sibling Git worktrees count through their shared Git directory. The warning does not stop the run. Unless--skip-install, the arena runspixel self-update --repo <repo> --build "cargo build --profile dev-release -p pixel-cli",pixel setup --yes,pixel build-index --history .,pixel doctor <repo> --fail-on yellow --json. Self-update and setup are fatal (a mise or Homebrewpixelis refused by self-update itself: the message is surfaced); the index and a yellow doctor are recorded, not fatal. The receipt recordspixel --version,command -v pixeland the repo HEAD. It rewires the home install to this build, as AGENTS.md "Optional Local Install Verification" describes; hand it back withpixel setupafterwards. - Grid in the operator's tab (
$HERDR_PANE_ID, which stays on the left with 30 % of the width): five cells on the right, three on top and two below, split sequentially (parallel Herdr splits race) with ratios 1/3, 1/2 for equal columns. Panes and agents are namedultima-<cli>; every created pane id is saved to$TMPDIR/ultima-arena-<uid>/last-run.jsonas it is made. - Order:
random.Random(seed).shuffle; agents are started and prompts are dispatched in that order, with no wait between turns. Anagent_pane_busystart is retried. - Dialogs: agy's one-time
Verbosity · esc to dismissbanner is dismissed with Esc. Codex's hook review (hooks need review,t trust all) is trusted only when every hook command in$CODEX_HOME/hooks.jsonand<repo>/.codex/hooks.jsonis Pixel's own<path>/pixel run-hook …; otherwise the CLI isneeds-operatorand no key is pressed. A folder-trust dialog is accepted (option number, then Enter); a tool approval is accepted only when the pending command is exactly the arena'spixel search-contentline. Anything else ends asblocked-dialogwith the screen excerpt. - Dispatch together, then monitor: prompts are sent to every ready CLI before waiting on any response. Pixel metrics may be suppressed when invocations overlap in one repository. Each relay is attributed only when that pane displays its invocation ID and the shared
actions.jsonlcontains the matching record; an overlapping timestamp alone is not attribution.NO-CALLuses that pane's displayed search command (or its matched relay evidence), never another CLI's action-log record. Run no other Pixel command in--repoduring the run. - Wait and judge: Pi uses its native fullscreen mode and a fresh run-owned
--session-dir. Its persistedpixel-briefcustom message proves injection; an exact bash tool call paired by ID with its tool result proves the command. A metrics box in that result must match the repo's action record. The receipt retains transcript paths, provider and model; assistant prose is never evidence. This also works when Pi's terminal viewport is scrolled to an older message. Pi completion comes from its terminal assistant event, so another CLI's action cannot finish Pi's turn. Claude starts with--verbose; when Codex finishes with hidden evidence, the runner opens its detailed transcript once with Ctrl+T. Neither step sends another model prompt.herdr agent promptruns without--wait(it reportsagent_prompt_stalledfor a prompt that ran), then pollagent getuntil the turn ended:done(Claude, Codex, agy, Devin) oridle(pi) after the state counter or the screen moved, then three polls of a still screen; Pi additionally needs its persisted completion event. 🟩 headers are also matched with line breaks joined, since a narrow pane hard-wraps them. Final scrollback is read from the exact pane withpane read --source recent-unwrapped --lines 2000; agent-level output may contain only the final answer. Wrappedprompt-submitandsearch-contenttokens are normalized before judging. - Receipt in
--out:receipt.json(identity, run number, seed, order, prompt, per CLI the evidence and verdicts, install steps),receipt.md(dated table),<cli>.scrollback.txt, and timestampedrun.log.eval/arena-results/README.mdindexes each dated run with its seed, order, counts, repository commit and dirty state;eval/.gitignorekeeps these artifacts out of Git. The last line printed isN/5 brief, M/5 metrics. A process lock refuses overlapping arena runs before pane changes or installation; default output paths include microseconds so a repeat never overwrites a previous receipt.
Statuses: done, evidence-complete (turn interrupted after both signals), timeout, start-failed, blocked-dialog, needs-operator. Per CLI the receipt also holds the
prompt-submit relay line (Claude, Codex, pi), the shell call count, and for metrics seen, occurrences and
outside_tool_framing (a box not inside │ ⎿ └ framing; best effort, a heuristic on the screen text).
Cleanup records each created pane's terminal identity. A Pi renderer crash may remove its agent binding and arena label; cleanup closes that shell only when the recorded terminal identity and specific renderer crash signature match, and preserves a pane assigned to another agent.
What it proves, and what it does not
- Proves: for this repo, binary and prompt, the hook wrote a brief that the host received (
partbytes plus displayed or persisted injection evidence), and the screen or host tool-result record contains a metrics box belonging to a real invocation of that CLI's turn. The receipt labels the evidence source; persisted context is not reported as on-screen visibility. - Does not prove the model used the brief, that a box is the relay rather than the shell tool's own output (look at
occurrencesandoutside_tool_framing), or anything about prompts other than this one. One prompt, one run per CLI: a FAIL is a lead to confirm withagent-session-debugging, not a verdict on a flaky run. PIXEL_BRIEF_LOG=0|false|offremoves the brief evidence: the run refuses to start.- The prompt's search path
crates/pixel/srcmust exist in--repo, so it runs against a pixel checkout.
Provider strings (crates/pixel/src/task_hook.rs TaskProvider::as_str)
claude, codex, pi, antigravity (the agy CLI), devin. Timestamps: ts in milliseconds on every brief-feedback.jsonl line, ts_ms on actions.jsonl.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |