Skip to content

ilandahan/AID / skills/pipeline-orchestrator

pipeline-orchestrator

Verified

Drives the Phase 4-5 dev pipeline state machine with sub-agent reviews and bounded autoresearch gates. Use when running /pipeline or executing an approved plan.

FileKindSize
SKILL.extended.mdmarkdown11.7 KB
SKILL.mdskill md70.3 KB
config.default.jsonconfig1.5 KB
gate.mjsscript38.3 KB
gate.test.mjsscript31.9 KB
references/escalation-protocol.mdmarkdown4.4 KB
references/load-bearing-assumptions.mdmarkdown3.2 KB
references/state-machine.mdmarkdown8.8 KB
test-escalation-loop.shexecutablescript13.8 KB

test-escalation-loop.sh

View on GitHub
#!/usr/bin/env bash
# End-to-end test of the escalation loop, in a throwaway git repo.
#   gate.mjs decides ESCALATE  ->  writes ESCALATION.json  ->  Stop hook BLOCKS
#   gate.mjs --resolve         ->  Stop hook stops blocking
# Nothing here touches a real project.
set -u
# TEST THE COPY THAT ACTUALLY RUNS (2026-09-04). Both GATE and the unit-test `cd` further down used
# to hardcode $HOME/.claude/skills/pipeline-orchestrator, so this whole end-to-end test validated an
# orphan copy that no session loads: Claude Code loads the skill from the AID plugin's versioned
# install path, and dashboard/qa/aid-pipeline-gate-live.mjs deliberately spawns this script with cwd
# set THERE — then line 49 cd'd away to the orphan anyway. A test pinned to the wrong artifact is a
# test that cannot fail when the artifact in use breaks. Resolved from this script's own location,
# captured BEFORE the `cd "$SANDBOX"` below, because $0 stops resolving once we leave this directory.
# WINDOWS-FORM path, not the MSYS one (measured 2026-09-04). Plain `pwd` in Git Bash returns a
# POSIX-style path like /c/dev's workspace/AID-clean/... and MSYS only auto-converts such arguments
# for a native binary when they contain no spaces. The plugin install path has none, so it converted
# and this script passed there; a source checkout under a directory with a space in its name (e.g.
# "dev's workspace") does not convert, so node received the POSIX path verbatim, resolved it against
# the current drive as C:\c\dev's..., and died MODULE_NOT_FOUND with exit 1. Every gate call then
# scored a FAIL for a reason that had nothing to do with the gate. `pwd -W` yields the native form
# (C:/dev's workspace/AID-clean/...), which node reads correctly; the fallback keeps this runnable
# on a real POSIX box.
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && { pwd -W 2>/dev/null || pwd; })"
GATE="$SKILL_DIR/gate.mjs"
# The hook stays at its installed path on purpose: that IS where Claude Code loads it from, so it is
# the real artifact under test rather than a copy.
HOOK="$HOME/.claude/hooks/dev-pipeline-gate.sh"

# Preflight. Every assertion below that "passes" does so on output NOT containing "block" — which is
# also what an empty string looks like. So a run where nothing executes at all still scores PASSes.
# That happened: the nightly manifest spawned a bare `bash`, which resolved to WSL's System32
# bash.exe, where node does not exist and HOME=/home/<user>. This script reported "4 passed, 8 failed"
# on a run in which the gate binary was never invoked once, and that mix read as a partial regression
# rather than as an environment that cannot test anything. Refuse to score instead.
command -v node >/dev/null 2>&1 || { echo "ABORT: no node on PATH (uname=$(uname -s), HOME=$HOME) — this environment cannot test the gate; not scoring any assertion"; exit 1; }
[ -f "$GATE" ] || { echo "ABORT: gate.mjs not found at $GATE (HOME=$HOME) — not scoring any assertion"; exit 1; }
[ -f "$HOOK" ] || { echo "ABORT: hook not found at $HOOK (HOME=$HOME) — not scoring any assertion"; exit 1; }

SANDBOX=$(mktemp -d)
pass=0; fail=0
ok()   { echo "  PASS  $1"; pass=$((pass+1)); }
bad()  { echo "  FAIL  $1"; echo "        got: $2"; fail=$((fail+1)); }

# "the hook did NOT block" — asserted so a CRASHED hook cannot score a pass.
# Four assertions here used `case "$out" in *'"decision":"block"'*) bad ;; *) ok ;; esac`, which scores a
# PASS on any output lacking the word "block" — including the empty string a hook that died on a missing
# jq, an unset variable or a bad path produces. The preflight above stops a wholly dead environment from
# scoring, but a hook broken for any other reason still banked four PASSes.
#
# Empty output alone is NOT the discriminator: measured, the real hook prints a {"systemMessage":...}
# line when it declines to block on a dirty repo, and prints NOTHING AT ALL on the stop_hook_active
# bypass — both legitimate, both exit 0. So require the exit code to be 0 as the positive evidence that
# the hook ran and made a decision, and treat a non-zero exit as the failure it is.
not_block() { # $1 = label, $2 = hook stdout, $3 = hook exit code
  case "$2" in
    *'"decision":"block"'*) bad "$1 (hook blocked)" "$2"; return ;;
  esac
  if [ "$3" != 0 ]; then
    bad "$1 (hook exited $3 — it FAILED rather than declined to block)" "${2:-<no output>}"
  else
    ok "$1"
  fi
}

cd "$SANDBOX" || exit 1

echo "== 0) unit tests for the gate decision function"
if unit=$( cd "$SKILL_DIR" && node --test 2>&1 ); then
  ok "gate.test.mjs unit tests pass"
else
  # Report the runner's own last lines, not a canned "go run it yourself". Discarding this output is
  # the same defect this script exists to catch: a red that names no cause.
  bad "gate.test.mjs unit tests FAILED" "$(printf '%s' "$unit" | tail -5)"
fi

git init -q . 2>/dev/null
git config user.email t@t; git config user.name t
mkdir -p .aid/pipeline
# rounds=5, threshold=9.5 — same shape as the real config
cat > .aid/pipeline/config.json <<'JSON'
{ "max_iterations": { "code_review": 5, "test_review": 5, "phase_gate_reexamine": 5, "visual_qa": 5 },
  "thresholds": { "code_review_pass": 9.5, "test_review_pass": 9.5, "visual_qa_pass": 7,
                  "auto_fail_on_critical_security": true } }
JSON
# 5 of 5 rounds already spent — the exact state that produced a false PASS at 7.9 historically
cat > .aid/pipeline/state.json <<'JSON'
{ "pipeline_status": "running", "current_step": "CODE_REVIEW",
  "iterations": { "code_review": 5, "test_review": 0, "phase_gate_reexamine": 0 } }
JSON
echo "console.log(1)" > app.js   # a changed source file, so the hook has work to consider
# state.json must be the NEWEST write, or the hook's own freshness rule blocks for a reason that has
# nothing to do with escalations — and cases 1 and 6 assert "does not block". They were only passing
# because app.js and state.json landed in the same second; two seconds apart the hook legitimately
# blocks and the baseline reports a failure that is really a test-setup artefact.
sleep 1
touch .aid/pipeline/state.json

echo "== 1) baseline: no escalation -> hook must NOT block"
out=$(printf '{}' | bash "$HOOK" 2>/dev/null); rc=$?
not_block "clean repo does not block" "$out" "$rc"

echo "== 2) gate: 7.9 with 5/5 rounds spent -> ESCALATE (exit 2) and writes ESCALATION.json"
node "$GATE" --step CODE_REVIEW --scores '{"overall":7.9,"security":7}' --root "$SANDBOX" >/dev/null 2>&1
rc=$?
[ "$rc" = 2 ] && ok "exit code 2 (ESCALATE)" || bad "expected exit 2, got $rc" "$rc"
[ -f .aid/pipeline/ESCALATION.json ] && ok "ESCALATION.json written" || bad "no escalation file" "missing"
grep -q '"status": "open"' .aid/pipeline/ESCALATION.json 2>/dev/null \
  && ok "escalation is open" || bad "escalation not open" "$(cat .aid/pipeline/ESCALATION.json 2>/dev/null | head -3)"

echo "== 3) Stop hook must now BLOCK the turn (this is the escalation reaching the human)"
out=$(printf '{}' | bash "$HOOK" 2>/dev/null)
case "$out" in
  *'"decision":"block"'*ESCALATED*) ok "hook blocks with the escalation reason" ;;
  *'"decision":"block"'*)           ok "hook blocks (reason text differs)" ;;
  *)                                bad "hook did NOT block on an open escalation" "$out" ;;
esac

echo "== 4) the loop guard still holds: stop_hook_active must never block (no infinite loop)"
out=$(printf '{"stop_hook_active":true}' | bash "$HOOK" 2>/dev/null); rc=$?
# measured: this path legitimately prints NOTHING and exits 0, which is why not_block keys on the exit
# code rather than on the presence of output
not_block "stop_hook_active bypasses (one interruption per turn)" "$out" "$rc"

echo "== 5) SKIP_PIPELINE_GATE=1 bypasses"
out=$(printf '{}' | SKIP_PIPELINE_GATE=1 bash "$HOOK" 2>/dev/null); rc=$?
not_block "SKIP_PIPELINE_GATE=1 bypasses" "$out" "$rc"

echo "== 6) resolving is the HUMAN's decision -> after --resolve the hook stops blocking"
node "$GATE" --resolve "accepted 7.9 for now; tracked as debt" --root "$SANDBOX" >/dev/null 2>&1
rc=$?
[ "$rc" = 0 ] && ok "--resolve exits 0" || bad "--resolve failed" "$rc"
out=$(printf '{}' | bash "$HOOK" 2>/dev/null); rc=$?
not_block "resolved escalation no longer blocks" "$out" "$rc"

echo "== 7) --check-escalation reports state for scripts"
node "$GATE" --check-escalation --root "$SANDBOX" >/dev/null 2>&1
rc=$?   # capture BEFORE the test: in `[ "$?" = 0 ] && ok || bad "..." "$?"` the second $? is the
        # exit of `[`, so the failure message reported 1 forever instead of the gate's real code.
[ "$rc" = 0 ] && ok "--check-escalation exit 0 when resolved" || bad "--check-escalation wrong exit" "$rc"

echo "== 8) a passing score never writes an escalation"
rm -f .aid/pipeline/ESCALATION.json
node "$GATE" --step CODE_REVIEW --scores '{"overall":9.7}' --root "$SANDBOX" >/dev/null 2>&1
rc=$?
{ [ "$rc" = 0 ] && [ ! -f .aid/pipeline/ESCALATION.json ]; } \
  && ok "9.7 passes and writes no escalation" || bad "9.7 mishandled (exit $rc)" "$rc"

echo "== 9) a dead agent leaves a record: --scores '{}' exits 3 AND grows step_history"
# The unit tests call recordDecision directly, so they pass under either ordering inside main().
# THIS is the assertion that pins the ordering: before 2026-09-04 the ERROR exit ran first, so a
# reviewer that produced nothing wrote no step_history row at all and the round was invisible.
# Relative to cwd, which IS $SANDBOX: an absolute MSYS /tmp/... path reaches Windows node verbatim
# and fails to resolve, and the `|| echo 0` would then report 0 both times — a green-looking
# no-change that proves nothing. hist() is the discriminator, so it must not be able to fail quietly.
hist() { node -e 'const s=require("./.aid/pipeline/state.json");console.log((s.step_history||[]).length)'; }
before=$(hist) || { echo "ABORT: cannot read step_history in $SANDBOX — case 9 cannot be scored"; exit 1; }
node "$GATE" --step CODE_REVIEW --scores '{}' --root "$SANDBOX" >/dev/null 2>&1
rc=$?
after=$(hist) || { echo "ABORT: cannot read step_history after the gate call"; exit 1; }
[ "$rc" = 3 ] && ok "a scoreless review exits 3 (ERROR), not 1 (FIX)" || bad "expected exit 3, got $rc" "$rc"
[ "$after" = "$((before+1))" ] && ok "the ERROR appended exactly one step_history row"   || bad "step_history went $before -> $after (expected $((before+1)))" "$after"
last=$(node -e 'const s=require("./.aid/pipeline/state.json");const h=s.step_history||[];console.log(h.length?h[h.length-1].result:"none")') || last=none
[ "$last" = "ERROR" ] && ok "the recorded row says ERROR" || bad "last step_history row is $last, not ERROR" "$last"

echo "== 10) --scores-b: the union caps the merged review, so a MAJOR cannot be averaged away"
# The pure-function tests cover mergeReviews. THIS covers the CLI path that reaches it, which is the
# part an orchestrator actually calls. A naive mean-of-overalls would report 7.8 and exit 1 for the
# WRONG reason, so assert the printed composite as well as the exit code.
rm -f .aid/pipeline/ESCALATION.json
cat > .aid/pipeline/state.json <<'JSON'
{ "pipeline_status": "running", "current_step": "CODE_REVIEW",
  "iterations": { "code_review": 0, "test_review": 0, "phase_gate_reexamine": 0 } }
JSON
CLEAN='{"overall":9.6,"security":9.6,"code_quality":9.6,"architecture":9.6,"documentation":9.6}'
FLAWED='{"overall":6.0,"security":6,"code_quality":6,"architecture":6,"documentation":6,"issues":[{"severity":"MAJOR","category":"code_quality","file":"a.ts","pattern":"silent catch"}]}'
out=$(node "$GATE" --step CODE_REVIEW --scores "$CLEAN" --scores-b "$FLAWED" --root "$SANDBOX" 2>&1); rc=$?
[ "$rc" = 1 ] && ok "one reviewer's MAJOR keeps the merged review in FIX" || bad "expected exit 1, got $rc" "$out"
case "$out" in
  *"capped at 6 by the UNION"*) ok "the cap came from the union, not from an average" ;;
  *)                            bad "no union cap in the output" "$out" ;;
esac
case "$out" in
  *7.8*) bad "the composite was averaged to 7.8 — a MAJOR was laundered" "$out" ;;
  *)     ok "the composite was not averaged past the cap" ;;
esac
out=$(node "$GATE" --step CODE_REVIEW --scores "$CLEAN" --scores-b '{"overall":9.4,"security":9.4,"code_quality":9.4,"architecture":9.4,"documentation":9.4}' --root "$SANDBOX" 2>&1); rc=$?
[ "$rc" = 0 ] && ok "two clean reviews merge to 9.5 and PASS" || bad "expected exit 0, got $rc" "$out"

echo "== 11) --brief-check refuses a brief wider than the cap, and refuses an undeclared one"
cat > .aid/pipeline/config.json <<'JSON'
{ "max_iterations": { "code_review": 5, "test_review": 5, "phase_gate_reexamine": 5, "visual_qa": 5 },
  "thresholds": { "code_review_pass": 9.5, "test_review_pass": 9.5, "visual_qa_pass": 7,
                  "auto_fail_on_critical_security": true, "max_surfaces": 2 } }
JSON
rm -f .aid/pipeline/ESCALATION.json
printf -- '---
task_id: t1
surfaces:
  - name: a
    path: p1
  - name: b
    path: p2
---
' > brief.ok.md
node "$GATE" --brief-check brief.ok.md --root "$SANDBOX" >/dev/null 2>&1; rc=$?
[ "$rc" = 0 ] && ok "two surfaces pass the cap" || bad "expected exit 0, got $rc" "$rc"
printf -- '---
task_id: t1
surfaces:
  - name: a
    path: p1
  - name: b
    path: p2
  - name: c
    path: p3
---
' > brief.wide.md
node "$GATE" --brief-check brief.wide.md --root "$SANDBOX" >/dev/null 2>&1; rc=$?
[ "$rc" = 2 ] && ok "three surfaces ESCALATE (exit 2)" || bad "expected exit 2, got $rc" "$rc"
[ -f .aid/pipeline/ESCALATION.json ] && ok "the width escalation reached the human" || bad "no escalation file" "missing"
printf -- '---
task_id: t1
---
' > brief.bare.md
node "$GATE" --brief-check brief.bare.md --root "$SANDBOX" >/dev/null 2>&1; rc=$?
# The load-bearing one: if an undeclared brief PASSED, skipping the declaration would be the cheapest
# way past the cap and the whole check would be decorative.
[ "$rc" = 3 ] && ok "an undeclared brief is ERROR (exit 3), not a pass" || bad "expected exit 3, got $rc" "$rc"
node "$GATE" --resolve "test cleanup" --root "$SANDBOX" >/dev/null 2>&1

echo
echo "RESULT: $pass passed, $fail failed"
cd /; rm -rf "$SANDBOX" 2>/dev/null
[ "$fail" = 0 ] || exit 1
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall