ccoalm/ccl-skills / skills/skill-extraction-workflow
skill-extraction-workflow
Flagged复盘 / 沉淀 / 总结经验 / 补进技能 / 技能缺陷 / 流程优化 / would other teammates hit this / will we forget next time / teamwide recurrence / extract skill / post-mortem / 深度 review·审计技能仓库 / 对标外部技能包找 gap(benchmark superpowers·gstack 等) / 核查全局安装点(~/.config/opencode 等)旧快照是否遮蔽本仓技能 / 本仓(ccl-skills 等共享技能仓)OpenCode 项目配置·命令治理 → turn reusable lessons into durable CCL skills.
scripts/test_eval_routing_bank_resolution.sh
View on GitHub#!/usr/bin/env bash
# Regression test for the bank runner's screening-vs-action resolution signal.
#
# A report taken below the action floor locates candidates; it does not license
# a description edit. The runner says so on stdout and in the JSON report, and
# this test pins BOTH sides of the number — the reference that states the rule
# and the executable that enforces it — so they cannot drift apart the way the
# description-length thresholds once did.
#
# Uses a fake `claude` earlier in PATH: never invokes a real CLI or account.
set -euo pipefail
# Ruby takes its encoding from the locale; under a POSIX/unset locale it reads the UTF-8 skill text as US-ASCII and crashes. Pin UTF-8, as CI runs.
case " ${RUBYOPT:-} " in *" -Ku "*) ;; *) export RUBYOPT="-Ku${RUBYOPT:+ $RUBYOPT}" ;; esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
EVAL_SCRIPT="$SCRIPT_DIR/eval-routing-bank.rb"
DOC="$SCRIPT_DIR/../references/eval-routing.md"
[ -f "$EVAL_SCRIPT" ] || { echo "FAIL: eval script not found: $EVAL_SCRIPT" >&2; exit 1; }
[ -f "$DOC" ] || { echo "FAIL: reference not found: $DOC" >&2; exit 1; }
fail() { echo "FAIL: $*" >&2; exit 1; }
assert_contains() { case "$2" in *"$1"*) : ;; *) fail "expected output to contain: $1${3:+ ($3)}";; esac; }
assert_absent() { case "$2" in *"$1"*) fail "expected output NOT to contain: $1${3:+ ($3)}";; *) : ;; esac; }
TMP="$(mktemp -d "${TMPDIR:-/tmp}/eval-routing-bank-resolution.XXXXXX")"
trap 'rm -rf "$TMP"' EXIT
REPO="$TMP/repo"
FAKE_BIN="$TMP/bin"
mkdir -p "$REPO/skills/testing-strategy" "$REPO/skills/tighten-doc" "$REPO/eval" "$FAKE_BIN"
git -C "$TMP" init -q repo
git -C "$REPO" config user.email test@example.invalid
git -C "$REPO" config user.name "Test User"
cat > "$REPO/skills/testing-strategy/SKILL.md" <<'EOF'
---
description: Use when choosing test layers and regression evidence.
---
# Testing Strategy
EOF
# Keeps the outcome space larger than {expected, none} so the fixture is not
# vacuous: a grader that always answered the only skill would pass trivially.
cat > "$REPO/skills/tighten-doc/SKILL.md" <<'EOF'
---
description: Use when polishing document wording.
---
# Tighten Doc
EOF
# TWO cases, deliberately: with one case a per-case verdict and a report-wide
# conjunction are indistinguishable, so every assertion below would also pass
# against the defect they exist to catch. The degraded fixture fails only the
# first utterance, which leaves one case short of the floor and one clear of it.
cat > "$REPO/eval/routing-tasks.jsonl" <<'EOF'
{"id":"resolution-fixture","utterance":"补一个回归测试","expected_skill":"testing-strategy","frozen_at_sha":""}
{"id":"resolution-fixture-b","utterance":"帮我润色这段文档","expected_skill":"tighten-doc","frozen_at_sha":""}
EOF
git -C "$REPO" add -A
git -C "$REPO" commit -qm "fixture"
cat > "$FAKE_BIN/claude" <<'EOF'
#!/usr/bin/env bash
prompt="$(cat)"
case "$prompt" in
*润色*) printf '{"selected_skill":"tighten-doc","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n' ;;
*) printf '{"selected_skill":"testing-strategy","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n' ;;
esac
EOF
chmod +x "$FAKE_BIN/claude"
export PATH="$FAKE_BIN:$PATH"
# --- (1) below the floor: banner printed, report says action_resolution false --
out_lo="$(ruby "$EVAL_SCRIPT" "$REPO" --replicas 3 --json "$TMP/lo.json" 2>&1)" \
|| fail "runner exited non-zero below the floor:\n$out_lo"
assert_contains "screening_resolution_only" "$out_lo" "sub-floor run must announce screening resolution"
assert_contains "replicas=3" "$out_lo" "banner must name the observed replica count"
assert_contains "floor 10" "$out_lo" "banner must name the required floor"
assert_contains "valid observations" "$out_lo" "banner must report the weakest task's valid-observation count, not the request alone"
grep -q '"action_resolution": false' "$TMP/lo.json" \
|| fail "sub-floor report must carry action_resolution:false — a consumer cannot read a printed banner"
# --- (2) at the floor: no banner, report says action_resolution true -----------
out_hi="$(ruby "$EVAL_SCRIPT" "$REPO" --replicas 10 --json "$TMP/hi.json" 2>&1)" \
|| fail "runner exited non-zero at the floor:\n$out_hi"
assert_absent "screening_resolution_only" "$out_hi" "a run at the floor must not be labelled screening-only"
grep -q '"action_resolution": true' "$TMP/hi.json" \
|| fail "at-floor report must carry action_resolution:true"
# --- (2a) the router model is part of the measurement -----------------------
# A run on the runner's default model screens; a description edit needs the tier
# that routes in use. The report says whether --model was given, the default run
# says so on stdout, and the reference states the rule the flag serves.
grep -q '"model_source": "default"' "$TMP/hi.json" \
|| fail "a run without --model must report model_source:default"
assert_contains "router_model_default" "$out_hi" "a run on the default router model must say so"
out_ex="$(ruby "$EVAL_SCRIPT" "$REPO" --replicas 1 --model fixture-router --json "$TMP/ex.json" 2>&1)" \
|| fail "runner exited non-zero with an explicit model:\n$out_ex"
grep -q '"model_source": "explicit"' "$TMP/ex.json" \
|| fail "a run with --model must report model_source:explicit"
assert_absent "router_model_default" "$out_ex" "an explicitly chosen router model must not be flagged as the default"
grep -q 'model_source: explicit' "$DOC" \
|| fail "eval-routing.md must state that a skill decision needs an explicitly chosen deploying-tier model"
# --- (2a') the router runs with hooks disabled --------------------------------
# Installed plugins' hooks would inject routing context the bank never asked for
# (the bootstrap is measured only through --with-bootstrap) and could rewrite the
# final answer, so the grader call must carry the hook-disabling settings.
cat > "$FAKE_BIN/claude" <<'EOF'
#!/usr/bin/env bash
printf '%s\n' "$@" > "$ROUTER_ARGS_FILE"
cat >/dev/null
printf '{"selected_skill":"testing-strategy","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n'
EOF
chmod +x "$FAKE_BIN/claude"
ROUTER_ARGS_FILE="$TMP/router-args" ruby "$EVAL_SCRIPT" "$REPO" --replicas 1 --json "$TMP/args.json" >/dev/null 2>&1 || true
grep -qF '"disableAllHooks":true' "$TMP/router-args" \
|| fail "the router call must disable hooks so plugin context cannot leak into the measurement"
# --- (2b) a nominal at-floor run with an invalid observation is NOT actionable -
# The floor is on valid observations. A grader that fails one call leaves the
# task below the floor while `--replicas` still reads 10, and a report that
# trusted the request would license an edit its evidence cannot support.
# The runner gives each unparsable verdict ONE retry, treating it as a sampling
# accident, so a fixture that fails a single call is repaired and records no
# error. The failure has to persist across the retry to leave the task short of
# the floor -- which is exactly the real shape this guards: a grader that is
# reliably unable to answer one utterance, not a stray quote.
cat > "$FAKE_BIN/claude" <<'EOF'
#!/usr/bin/env bash
prompt="$(cat)"
case "$prompt" in
*润色*) printf '{"selected_skill":"tighten-doc","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n'; exit 0 ;;
esac
echo x >> "$RESOLUTION_COUNT_FILE"
n=$(wc -l < "$RESOLUTION_COUNT_FILE" | tr -d ' ')
if [ "$n" = "3" ] || [ "$n" = "4" ]; then printf 'not json at all\n'; exit 0; fi
printf '{"selected_skill":"testing-strategy","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n'
EOF
chmod +x "$FAKE_BIN/claude"
: > "$TMP/count"
out_deg="$(RESOLUTION_COUNT_FILE="$TMP/count" ruby "$EVAL_SCRIPT" "$REPO" --replicas 10 --json "$TMP/deg.json" 2>&1)" \
|| fail "runner exited non-zero on the degraded run:\n$out_deg"
assert_contains "screening_resolution_only" "$out_deg" "a nominal 10-replica run with an invalid observation must not be actionable"
grep -q '"action_resolution": false' "$TMP/deg.json" \
|| fail "a 10-replica run with only 9 valid observations must report action_resolution:false"
grep -q '"min_valid_observations": 9' "$TMP/deg.json" \
|| fail "the report must expose the weakest task's valid-observation count"
# --- (2c) the verdict is per case, and says nothing about a case not measured --
# A report-wide boolean alone is unsound in the licensing direction: a subset run
# over one case would otherwise read as licence for an edit to a case the run
# never graded. Each result carries its own actionable and valid_observations.
grep -q '"actionable": true' "$TMP/hi.json" \
|| fail "an at-floor case must carry its own actionable:true"
grep -q '"valid_observations": 10' "$TMP/hi.json" \
|| fail "each case must expose its own valid-observation count"
grep -q '"action_resolution_scope"' "$TMP/hi.json" \
|| fail "the report must say its top-level verdict covers only the cases it measured"
grep -q '"actionable": false' "$TMP/deg.json" \
|| fail "a case left short by an invalid observation must carry actionable:false"
# The split is the point: the degraded case is short, the other one is not, and a
# report-wide conjunction substituted for the per-case field would mark both false.
python3 - "$TMP/deg.json" <<'PYEOF' || fail "the healthy case must stay actionable while the degraded one does not"
import json,sys
d=json.load(open(sys.argv[1]))
by={r["id"]:r for r in d["results"]}
ok = (by["resolution-fixture"]["actionable"] is False
and by["resolution-fixture-b"]["actionable"] is True
and by["resolution-fixture-b"]["valid_observations"] == 10
and d["action_resolution"] is False)
sys.exit(0 if ok else 1)
PYEOF
# --- (2d) shapes where a result could carry no usable verdict at all ----------
# The resolution loop reads every result's verdict list. Review could not check
# from the bounded packet that a list is always present, so the shapes that would
# expose an absent one are pinned here: a single-replica run, and a task whose
# every replica fails. Neither may crash, and neither may report itself
# actionable.
: > "$TMP/count"
out_one="$(RESOLUTION_COUNT_FILE="$TMP/count" ruby "$EVAL_SCRIPT" "$REPO" --replicas 1 --json "$TMP/one.json" 2>&1)" \
|| fail "runner exited non-zero on a single-replica run:\n$out_one"
grep -q '"action_resolution": false' "$TMP/one.json" \
|| fail "a single-replica run is below the floor and must not be actionable"
python3 - "$TMP/one.json" <<'PYEOF' || fail "every result of a single-replica run must carry its own verdict list and count"
import json,sys
d=json.load(open(sys.argv[1]))
ok = bool(d["results"]) and all(
isinstance(r.get("verdicts"), list) and len(r["verdicts"]) == 1
and r["valid_observations"] <= 1 and r["actionable"] is False
for r in d["results"])
sys.exit(0 if ok else 1)
PYEOF
cat > "$FAKE_BIN/claude" <<'EOF'
#!/usr/bin/env bash
cat >/dev/null
printf 'never json\n'
EOF
chmod +x "$FAKE_BIN/claude"
# Every task fails every replica: the runner reports the wholly-unmeasured run and
# exits 3 by its documented contract, and the report must still be well formed.
# `set -e` would kill the suite on the non-zero exit before it could be read, so
# the status is captured in the same command that produces it.
rc=0
ruby "$EVAL_SCRIPT" "$REPO" --replicas 2 --json "$TMP/allfail.json" >/dev/null 2>&1 || rc=$?
[ "$rc" = "3" ] || fail "a wholly failed run must exit 3 by the runner's contract, got $rc"
python3 - "$TMP/allfail.json" <<'PYEOF' || fail "a wholly failed run must still report zero valid observations and no actionable case"
import json,sys
d=json.load(open(sys.argv[1]))
ok = bool(d["results"]) and d["action_resolution"] is False and d["min_valid_observations"] == 0 \
and all(r["valid_observations"] == 0 and r["actionable"] is False and isinstance(r["verdicts"], list)
for r in d["results"])
sys.exit(0 if ok else 1)
PYEOF
# --- (2e) a parseable answer naming no catalog skill is not an observation ------
# Review could not see from the packet whether a well-formed JSON verdict whose
# selected_skill is not in the catalog counts toward the floor. It must not: it
# is absence of evidence, not evidence against the route, so it is an ERROR
# verdict and the case it belongs to stays short of the floor.
cat > "$FAKE_BIN/claude" <<'EOF'
#!/usr/bin/env bash
cat >/dev/null
printf '{"selected_skill":"skill-that-does-not-exist","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n'
EOF
chmod +x "$FAKE_BIN/claude"
rc=0
ruby "$EVAL_SCRIPT" "$REPO" --replicas 10 --json "$TMP/badsel.json" >/dev/null 2>&1 || rc=$?
python3 - "$TMP/badsel.json" <<'PYEOF' || fail "a parseable verdict naming a non-catalog skill must count as an error, not a usable observation"
import json,sys
d=json.load(open(sys.argv[1]))
ok = d["action_resolution"] is False and d["min_valid_observations"] == 0 \
and all(r["valid_observations"] == 0 and r["actionable"] is False for r in d["results"]) \
and all(v["status"] == "ERROR" for r in d["results"] for v in r["verdicts"])
sys.exit(0 if ok else 1)
PYEOF
# --- (2f) a skill the prompt never offered is not selectable ---------------------
# A directory with a SKILL.md but no usable description is filtered out of the
# prompt catalog. If the selectable-name list were built separately it could still
# admit that name, and a verdict naming it would count. Both must come from one
# filtered list, so selecting the unoffered skill is an ERROR like any other
# non-catalog name.
mkdir -p "$REPO/skills/ghost-skill"
printf -- '---\ndescription: ""\n---\n# Ghost\n' > "$REPO/skills/ghost-skill/SKILL.md"
git -C "$REPO" add -A && git -C "$REPO" commit -qm "ghost skill with empty description"
cat > "$FAKE_BIN/claude" <<'EOF'
#!/usr/bin/env bash
cat >/dev/null
printf '{"selected_skill":"ghost-skill","clarify":false,"confidence":0.9,"rationale_short":"fixture"}\n'
EOF
chmod +x "$FAKE_BIN/claude"
rc=0
ruby "$EVAL_SCRIPT" "$REPO" --replicas 10 --json "$TMP/ghost.json" >/dev/null 2>&1 || rc=$?
python3 - "$TMP/ghost.json" <<'PYEOF' || fail "a verdict naming a skill the prompt never offered must be an error, not a usable observation"
import json,sys
d=json.load(open(sys.argv[1]))
ok = d["action_resolution"] is False and all(v["status"]=="ERROR" for r in d["results"] for v in r["verdicts"])
sys.exit(0 if ok else 1)
PYEOF
# --- (3) the two sides of the number must agree -------------------------------
# The rule is only as good as the agreement between the reference that states it
# and the executable that enforces it. Compare the numbers themselves rather
# than pinning one literal in two places.
exe_min="$(grep -oE 'ACTION_RESOLUTION_MIN_REPLICAS = [0-9]+' "$EVAL_SCRIPT" | grep -oE '[0-9]+$' | sort -u)"
[ -n "$exe_min" ] || fail "could not read ACTION_RESOLUTION_MIN_REPLICAS from $EVAL_SCRIPT"
[ "$(printf '%s\n' "$exe_min" | wc -l | tr -d ' ')" = "1" ] \
|| fail "ACTION_RESOLUTION_MIN_REPLICAS is defined with more than one value: $exe_min"
doc_min="$(grep -oE 'replicas < [0-9]+' "$DOC" | grep -oE '[0-9]+$' | sort -u)"
[ -n "$doc_min" ] || fail "$DOC does not state the replicas floor as \`replicas < N\`"
[ "$doc_min" = "$exe_min" ] \
|| fail "doc states floor $doc_min but the runner enforces $exe_min — they must be one number"
# The prose obligation (">=N valid observations before acting") must name the
# same number too, so a reader following the sentence and a consumer reading the
# report are held to one bar.
grep -qF "≥${exe_min} 个有效观测" "$DOC" \
|| fail "$DOC must state the per-case obligation with the same floor (≥${exe_min} 个有效观测)"
echo "PASS: eval-routing-bank resolution signal (banner, report field, doc/executor agreement)"
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |