Skip to content

ccoalm/ccl-skills / skills/skill-extraction-workflow

skill-extraction-workflow

Flagged

复盘 / 沉淀 / 总结经验 / 补进技能 / 技能缺陷 / 流程优化 / would other teammates hit this / will we forget next time / teamwide recurrence / extract skill / post-mortem / 深度 review·审计技能仓库 / 对标外部技能包找 gap(benchmark superpowers·gstack 等) / 核查全局安装点(~/.config/opencode 等)旧快照是否遮蔽本仓技能 / 本仓(ccl-skills 等共享技能仓)OpenCode 项目配置·命令治理 → turn reusable lessons into durable CCL skills.

FileKindSize
SKILL.mdskill md111.0 KB
agents/openai.yamlconfig257 B
references/analysis-parse-fix-test-challenge-replay.mdmarkdown1.5 KB
references/attention-budget-ratchet.mdmarkdown13.7 KB
references/attribution-verification.mdmarkdown4.4 KB
references/bootstrap-slim-c3-obligation-table.mdmarkdown8.0 KB
references/correction-routing-map.mdmarkdown3.7 KB
references/coverage-exhaustion-traps.mdmarkdown19.6 KB
references/description-authoring.mdmarkdown21.1 KB
references/dual-track-review-gate.mdmarkdown110.6 KB
references/eval-routing.mdmarkdown22.2 KB
references/evidence-card-template.mdmarkdown3.2 KB
references/example-domain-preselect.mdmarkdown4.6 KB
references/external-practice-controls.mdmarkdown45.5 KB
references/extraction-lifecycle-handoff.mdmarkdown6.9 KB
references/extraction-quickstart.mdmarkdown20.2 KB
references/firing-point-placement.mdmarkdown21.3 KB
references/harness-patterns-and-eval.mdmarkdown42.6 KB
references/incident-postmortem-extraction.mdmarkdown28.8 KB
references/l0-l1-l2-routing.mdmarkdown10.3 KB
references/online-skill-review.mdmarkdown6.7 KB
references/parallel-stack-references-pattern.mdmarkdown35.9 KB
references/r0-leakage-audit.mdmarkdown19.9 KB
references/recurring-anti-patterns-checklist.mdmarkdown34.4 KB
references/resume-paused-delivery.mdmarkdown3.0 KB
references/review-feedback-mining.mdmarkdown5.7 KB
references/review-finding-standards.mdmarkdown7.3 KB
references/review-rubric.mdmarkdown3.1 KB
references/rule-consolidation.mdmarkdown38.2 KB
references/skill-listing-budget.mdmarkdown1.7 KB
references/source-register.mdmarkdown1.0 MB
references/source-to-skill-extraction.mdmarkdown109.3 KB
references/two-source-extraction-pattern.mdmarkdown19.4 KB
references/uiux-judgment-extraction.mdmarkdown19.1 KB
references/uiux-routing-map.mdmarkdown5.8 KB
references/validation-and-landing.mdmarkdown46.9 KB
scripts/AGENTS.mdmarkdown635 B
scripts/check-ccl-skills.shexecutablescript100.7 KB
scripts/check-contract-anchors.shexecutablescript5.6 KB
scripts/check-evidence-card-leak.shexecutablescript26.1 KB
scripts/check-mr-target-freshness.shexecutablescript9.4 KB
scripts/check-size-budget.shexecutablescript36.0 KB
scripts/check-sync-pointers.shexecutablescript22.6 KB
scripts/check_review_evidence_present.pyexecutablescript4.6 KB
scripts/contract-anchors.tsvother3.8 KB
scripts/entrypoint_form_census.pyexecutablescript7.1 KB
scripts/eval-golden-trace.rbexecutablescript10.0 KB
scripts/eval-health.rbexecutablescript14.6 KB
scripts/eval-routing-bank.rbexecutablescript30.4 KB
scripts/eval-routing.rbexecutablescript9.2 KB
scripts/extraction_review_gate.shexecutablescript1.8 KB
scripts/gate_receipt.pyexecutablescript24.7 KB
scripts/generic-r0-leak-scan.shexecutablescript12.4 KB
scripts/governing-chain-diff.pyexecutablescript12.8 KB
scripts/impact-chain-gate.rbexecutablescript124.4 KB
scripts/obligation-ledger.pyscript107.0 KB
scripts/reference-access-census.shexecutablescript8.3 KB
scripts/register-firing-path-resolution.rbexecutablescript48.4 KB
scripts/shared_git_surface_gate.pyexecutablescript38.0 KB
scripts/skill-behavior-eval.pyexecutablescript39.3 KB
scripts/skill-paired-eval.pyexecutablescript76.9 KB
scripts/source-register-lifecycle.rbscript1.5 KB
scripts/source-register-pending-status.rbscript2.0 KB
scripts/test_ai_coding_implementation_gates.shexecutablescript85.2 KB
scripts/test_antipattern_grep_panel.shexecutablescript3.2 KB
scripts/test_body_compliance_grading.shexecutablescript21.0 KB
scripts/test_check_ccl_impact_chain_refscripts.shexecutablescript145.1 KB
scripts/test_check_ccl_r0_status.shexecutablescript4.2 KB
scripts/test_check_ccl_register_pending_exclusion.shexecutablescript7.4 KB
scripts/test_check_ccl_regressions.shexecutablescript14.3 KB
scripts/test_check_ccl_route_drift.shexecutablescript22.6 KB
scripts/test_check_ccl_size_budget.shexecutablescript59.5 KB
scripts/test_check_ccl_skill_catalog.shexecutablescript23.3 KB
scripts/test_check_ccl_source_register_lifecycle.shexecutablescript6.1 KB
scripts/test_check_contract_anchors.shexecutablescript7.4 KB
scripts/test_check_mr_target_freshness.shexecutablescript12.8 KB
scripts/test_check_review_evidence_present.shexecutablescript4.3 KB
scripts/test_check_sync_pointers.shexecutablescript32.5 KB
scripts/test_ci_checkout_ref_binding.shexecutablescript8.2 KB
scripts/test_controlled_escalation_pins.shexecutablescript7.7 KB
scripts/test_entrypoint_domain_scan_terms.shexecutablescript8.8 KB
scripts/test_entrypoint_form_census.shexecutablescript6.3 KB
scripts/test_eval_routing_bank_grader_diagnostics.shexecutablescript20.6 KB
scripts/test_eval_routing_bank_resolution.shexecutablescript15.3 KB
scripts/test_eval_routing_bank_surface_binding.shexecutablescript6.9 KB
scripts/test_eval_routing_prose_target.shexecutablescript3.7 KB
scripts/test_eval_runtime.pyexecutablescript43.3 KB
scripts/test_extraction_review_gate.shexecutablescript15.2 KB
scripts/test_frozen_case_sanctity.shexecutablescript6.9 KB
scripts/test_frozen_case_sanctity_selfproof.shexecutablescript4.6 KB
scripts/test_gate_receipt.shexecutablescript17.7 KB
scripts/test_generic_r0_leak_scan.shexecutablescript6.4 KB
scripts/test_git_identity_predicate_gate.shscript13.8 KB
scripts/test_governing_chain_diff.shexecutablescript15.1 KB
scripts/test_impact_chain_gate_dateless_host.shexecutablescript6.8 KB
scripts/test_impact_chain_gate_verdict_differential.shexecutablescript20.8 KB
scripts/test_impact_chain_round_attribution.shexecutablescript32.5 KB
scripts/test_impact_chain_self_adjudication.shexecutablescript31.4 KB
scripts/test_impact_chain_source_refuted.shexecutablescript14.7 KB
scripts/test_liveness_predicate_gate.shexecutablescript14.5 KB
scripts/test_locale_independent_gates.shexecutablescript13.0 KB
scripts/test_obligation_ledger.shexecutablescript94.9 KB
scripts/test_obligation_ledger_repo_audit.shexecutablescript2.7 KB
scripts/test_pinned_phrase_mutation_walk.shexecutablescript6.5 KB
scripts/test_reference_access_census.shexecutablescript17.3 KB
scripts/test_register_firing_path_resolution.shexecutablescript41.0 KB
scripts/test_register_firing_path_wiring.shexecutablescript29.9 KB
scripts/test_regression_runner_lanes.shexecutablescript4.5 KB
scripts/test_regression_runner_registration.shexecutablescript1.5 KB
scripts/test_routing_bank_integrity.shexecutablescript15.5 KB
scripts/test_routing_pointer_integrity.shexecutablescript18.9 KB
scripts/test_shared_git_surface_gate.shexecutablescript76.8 KB
scripts/test_skill_paired_eval.pyscript60.0 KB
scripts/test_teardown_guard_pins.shexecutablescript17.0 KB
scripts/test_uiux_delivery_contract.shexecutablescript118.6 KB
scripts/test_uiux_loading_budget.shexecutablescript15.4 KB
scripts/test_validate_skill_credential_cwd.shexecutablescript2.6 KB
scripts/test_validate_skill_cross_refs.shexecutablescript4.8 KB
scripts/test_validate_skill_root_depth.shexecutablescript2.5 KB
scripts/validate-skill.shexecutablescript10.5 KB

scripts/eval-routing-bank.rb

View on GitHub
#!/usr/bin/env ruby
# frozen_string_literal: true

# F4 Tier-2 routing compatibility eval (cheap-LLM grader over a frozen task bank).
#
# Measures whether the current skill descriptions make a cheap model route a fixed
# set of utterances to the expected skill. This is a ROUTING-COMPATIBILITY signal,
# NOT a truth oracle — the grader can be wrong. Advisory dashboard: it never blocks
# a merge and is not wired into check-ccl-skills.sh.
#
# Anti-game: each task carries frozen_at_sha; `root` resolves to the repository
# root commit, while an explicit SHA must be an ancestor of HEAD. The runner also
# flags when the same change touches both the bank and skill
# descriptions (so "fix skill + rewrite test to pass" is visible).
#
# Usage:
#   eval-routing-bank.rb <repo-root> [--bank <path>] [--model <name>] [--limit N]
#                        [--dry-run] [--json <path>] [--baseline <path>] [--timeout S]
#                        [--desc-budget-chars N] [--with-bootstrap] [--replicas N]
#
# --desc-budget-chars N simulates a consumer that truncates each skill description
# to its first N characters before routing (e.g. Codex compresses the skill listing
# under a ~2%-of-context budget, 8000 chars when the window is unknown, shortening
# descriptions first). Run the bank once plain and once with a budget to see which
# routes only survive on the description tail — those triggers need front-loading.
#
# expected_skill "none" marks negative controls (out-of-library utterances) and
# coverage-gap probes (in-domain utterances no skill owns): the correct outcome
# is rejection, and a catalog skill claiming them is labeled "absorbed".
# --replicas N grades each task N times: task status is the conservative
# consensus (every observed replica must PASS), replica top1 agreement is
# reported, and disagreeing replicas label the task "ownership_split". clarify
# and low-confidence counts are first-class report fields either way.
# Reports from different (bank, replicas) configurations are different rulers —
# do not diff them as a regression signal.
# Exit: 0 = ran (advisory); 2 = usage error; 3 = grader entirely unavailable.

require "yaml"
require "json"
require "open3"
require "shellwords"
require "timeout"
require "digest"

# Resolution floor for ACTING on a measured case. A report taken below it
# LOCATES candidates; it does not license a description edit, because at three
# replicas a case that routes correctly 80-90% of the time reads as failing and
# a one-off deviation reads as a finding. `references/eval-routing.md` owns the
# rule and states the same number; test_eval_routing_bank_resolution.sh pins the
# two sides together so they cannot drift apart.
ACTION_RESOLUTION_MIN_REPLICAS = 10

def arg(flag, default = nil)
  i = ARGV.index(flag)
  i ? ARGV[i + 1] : default
end

root = ARGV[0]
if root.nil? || root.start_with?("-")
  warn "usage: eval-routing-bank.rb <repo-root> [--bank p] [--model m] [--limit N] [--dry-run] [--json p] [--baseline p] [--timeout S] [--desc-budget-chars N] [--replicas N]"
  exit 2
end
bank_path = arg("--bank", File.join(root, "eval", "routing-tasks.jsonl"))
model = arg("--model", "claude-haiku-4-5")
# Routing is decided by the model that reads the skill listing. The default is a
# cheap screen; record whether the router model was chosen on purpose.
model_source = ARGV.include?("--model") ? "explicit" : "default"
limit = (l = arg("--limit")) ? l.to_i : nil
dry_run = ARGV.include?("--dry-run")
json_path = arg("--json")
baseline_path = arg("--baseline")
desc_budget = nil
if ARGV.include?("--desc-budget-chars")
  if ARGV.count("--desc-budget-chars") > 1
    warn "--desc-budget-chars given more than once; pass it a single time"
    exit 2
  end
  b = arg("--desc-budget-chars")
  # Strict: a missing value (flag last / next token is another flag) or a
  # non-integer must fail loudly, or the truncation arm silently becomes the
  # baseline arm (to_i coerces "200x" -> 200, nil -> 0).
  unless b && b =~ /\A[1-9]\d*\z/
    warn "--desc-budget-chars requires a positive integer value, got #{b.inspect}"
    exit 2
  end
  desc_budget = b.to_i
end
timeout_s = (t = arg("--timeout")) ? t.to_i : 60
replicas = 1
if ARGV.include?("--replicas")
  r = arg("--replicas")
  # Same strictness as --desc-budget-chars: a silent to_i coercion would turn a
  # typo into "1 replica" and the agreement metric would quietly measure nothing.
  unless r && r =~ /\A[1-9]\d*\z/
    warn "--replicas requires a positive integer value, got #{r.inspect}"
    exit 2
  end
  replicas = r.to_i
end

unless File.file?(bank_path)
  warn "eval_bank_missing: #{bank_path}"
  exit 2
end

# ONE immutable snapshot of the bank: the tasks are parsed from, and every
# bank-derived hash below is computed from, the SAME bytes — so grading and the
# report's self-identification cannot diverge if the file changes mid-run.
bank_bytes = File.binread(bank_path)
tasks = []
bank_bytes.dup.force_encoding(Encoding::UTF_8).each_line.with_index(1) do |line, n|
  line = line.strip
  next if line.empty?
  begin
    tasks << JSON.parse(line)
  rescue JSON::ParserError => e
    warn "eval_bank_parse_error: line #{n}: #{e.message}"
    exit 2
  end
end
tasks = tasks.first(limit) if limit

# Schema validation: a task must be answerable, and must not be self-contradictory.
# expected_skill "none" is the negative-control / coverage-gap sentinel: the
# correct routing outcome is that NO catalog skill claims the utterance.
all_outcomes = Dir[File.join(root, "skills", "*", "SKILL.md")]
               .map { |p| File.basename(File.dirname(p)) } + ["none"]
tasks.each do |t|
  id = t["id"] || "(no id)"
  if t["utterance"].to_s.strip.empty? || t["expected_skill"].to_s.strip.empty?
    warn "eval_bank_invalid_task: #{id}: utterance and expected_skill are required"
    exit 2
  end
  # Type-check the ORIGINAL values FIRST: a present-but-non-list field (e.g. ""
  # or a bare string or a number) must fail as a usage error before any
  # membership check touches it — Ruby strings are truthy and respond to
  # include? (substring semantics), and non-strings would raise a bare
  # NoMethodError instead of the documented invalid-bank diagnostic.
  %w[must_not_route_to acceptable].each do |f|
    next unless t.key?(f)
    unless t[f].is_a?(Array)
      warn "eval_bank_invalid_task: #{id}: #{f} must be a list, got #{t[f].inspect}"
      exit 2
    end
  end
  if (t["must_not_route_to"] || []).include?(t["expected_skill"])
    warn "eval_bank_invalid_task: #{id}: expected_skill is also in must_not_route_to (impossible)"
    exit 2
  end
  if (t["must_not_route_to"] || []).include?("none")
    warn "eval_bank_invalid_task: #{id}: \"none\" is a sentinel outcome, not a routable target for must_not_route_to"
    exit 2
  end
  # Optional acceptable[] names defensible alternate outcomes (a skill name or
  # "none") for utterances with more than one correct route — e.g. a coverage-gap
  # ask where both coordinator intake and rejection are right. It must not
  # restate expected_skill or contradict must_not_route_to.
  acc = t["acceptable"] || []
  if acc.include?(t["expected_skill"])
    warn "eval_bank_invalid_task: #{id}: acceptable restates expected_skill"
    exit 2
  end
  unless (acc & (t["must_not_route_to"] || [])).empty?
    warn "eval_bank_invalid_task: #{id}: acceptable and must_not_route_to overlap (contradictory)"
    exit 2
  end
  # Anti-gaming: expected + acceptable must leave at least one outcome that
  # would FAIL the row. A row covering the complete catalog-and-none outcome
  # space passes on every valid grader selection — a vacuous fixture that fakes
  # green regardless of description behavior.
  if (all_outcomes - ([t["expected_skill"]] + acc)).empty?
    warn "eval_bank_invalid_task: #{id}: expected_skill plus acceptable cover every possible outcome (vacuous row)"
    exit 2
  end
end

# --- anti-game ---------------------------------------------------------------
def ancestor?(root, sha)
  return false if sha.nil? || sha.empty?
  system("git", "-C", root, "merge-base", "--is-ancestor", sha, "HEAD",
         out: File::NULL, err: File::NULL)
end

# Working-tree co-change: does the branch diff touch BOTH the bank and any
# SKILL.md description? (advisory governance warning). If the base ref cannot be
# resolved the check is UNAVAILABLE — surfaced explicitly, never silently false.
base = ENV["CCL_SKILL_BASE_REF"].to_s.strip
base = "origin/main" if base.empty?
changed, diff_status = Open3.capture2e("git", "-C", root, "diff", base, "HEAD", "--name-only")
co_change_check_ok = diff_status.success?
changed_files = co_change_check_ok ? changed.lines.map(&:strip) : []
bank_changed = changed_files.any? { |f| f.include?("routing-tasks.jsonl") }
desc_changed = changed_files.any? { |f| f.end_with?("/SKILL.md") }
co_change = bank_changed && desc_changed

# --- build skill routing surface (the same descriptions the agent routes on) -
# One filtered list feeds BOTH the prompt and the set of selectable names, so a
# skill the prompt never offered (no frontmatter, empty description) cannot be a
# valid selection: review noted the two were built by separate transformations
# whose filtering could drift apart, and a name allowed but never shown is exactly
# the shape that drift would let a verdict claim.
# map + compact rather than filter_map: the runner has to work on the oldest
# Ruby a host ships (macOS system Ruby is 2.6), and filter_map is 2.7+.
catalog_entries = Dir[File.join(root, "skills", "*", "SKILL.md")].sort.map do |path|
  name = File.basename(File.dirname(path))
  m = File.read(path).match(/\A---\s*\n(.*?)\n---\s*\n/m)
  next unless m
  desc = (YAML.safe_load(m[1]) rescue {})["description"].to_s.strip
  next if desc.empty?
  desc = desc[0, desc_budget] if desc_budget && desc.length > desc_budget
  [name, "### #{name}\n#{desc}"]
end.compact
catalog = catalog_entries.map(&:last).join("\n\n")
# The set of names a verdict may legitimately select; "none" is accepted separately.
catalog_names = catalog_entries.map(&:first)

# --- optional always-on entry-routing layer -----------------------------------
# The catalog above is the description-only surface. Hosts ALSO inject
# agent-context/session-start.md into every session, and that layer carries
# cross-skill discrimination rules the descriptions do not. Without this flag the
# bank cannot tell whether a routing miss is fixed by the injected layer, so every
# "this skill must be in the always-on layer" claim stayed unfalsifiable — and the
# layer has a hard zero-net-growth budget, which makes that claim expensive.
# Run the bank twice (plain, then --with-bootstrap) to measure what the layer buys.
bootstrap_layer = nil
if ARGV.include?("--with-bootstrap")
  bpath = File.join(root, "agent-context", "session-start.md")
  unless File.file?(bpath)
    warn "eval-routing-bank: --with-bootstrap given but #{bpath} is missing"
    exit 2
  end
  bootstrap_layer = File.read(bpath)[/ccl:entry-routing:start -->(.*?)<!-- ccl:entry-routing:end/m, 1]&.strip
  if bootstrap_layer.nil? || bootstrap_layer.empty?
    warn "eval-routing-bank: --with-bootstrap given but no ccl:entry-routing region in #{bpath}"
    exit 2
  end
end

# --- routing-surface self-identification -------------------------------------
# Every emitted report names the exact surface it graded, so a round artifact
# can never be attributed to the wrong wording by operator assertion alone.
# descriptions_sha256 mirrors the binding wrapper's surface_hash byte-for-byte
# (dir-name-tagged RAW description lines of every skills/*/SKILL.md in sorted
# glob order — grep semantics, YAML unparsed, trailing newline included — plus
# the bank file bytes), so a sidecar recomputing the same quantity externally
# is an independent check of this claim. per_skill maps each description line
# to its own hash; catalog_sha256 identifies the GRADED catalog text (post
# YAML-parse, post desc-budget truncation), which is what the model actually
# saw. Grading semantics are untouched: nothing below feeds the prompt or the
# verdicts.
surface_parts = +"".b
per_skill_desc_sha = {}
Dir[File.join(root, "skills", "*", "SKILL.md")].sort.each do |path|
  name = File.basename(File.dirname(path))
  line = File.open(path, "rb") { |f| f.each_line.find { |l| l.start_with?("description:") } }
  next unless line
  surface_parts << name.b << "\t".b << line.b
  per_skill_desc_sha[name] = Digest::SHA256.hexdigest(line)
end
routing_surface = {
  descriptions_sha256: Digest::SHA256.hexdigest(surface_parts + bank_bytes),
  per_skill_description_line_sha256: per_skill_desc_sha,
  catalog_sha256: Digest::SHA256.hexdigest(catalog),
  bank_sha256: Digest::SHA256.hexdigest(bank_bytes),
  bootstrap_layer_sha256: bootstrap_layer ? Digest::SHA256.hexdigest(bootstrap_layer) : nil
}

def build_prompt(catalog, utterance, bootstrap_layer = nil)
  <<~PROMPT
    You are a routing classifier for a CCL engineering skill system. Given a
    user utterance and the catalog of skills (name + description), pick the SINGLE
    best skill that should handle it. Use only the catalog; route by the
    descriptions' Use-when / Proactively / Skip rules.
    If NO skill's description covers the utterance, answer "none" — do not
    force-fit the nearest neighbor. Set "clarify" to true only when you would
    need to ask the user a clarifying question before committing to a route.

    Output ONLY a JSON object on one line, no other text:
    {"selected_skill": "<exact skill name from the catalog, or none>", "clarify": <true|false>, "confidence": <0.0-1.0>, "rationale_short": "<one short clause>"}

    #{bootstrap_layer ? "== ALWAYS-ON ENTRY ROUTING LAYER (injected into every session; takes precedence for entry routing) ==\n#{bootstrap_layer}\n" : ""}
    == SKILL CATALOG ==
    #{catalog}

    == UTTERANCE ==
    #{utterance}
  PROMPT
end

# Extract the first BALANCED top-level JSON object (string/escape aware) so extra
# braces, markdown fences, or trailing prose after the object do not corrupt it.
def extract_json(text)
  start = text.index("{")
  return nil unless start
  depth = 0
  in_str = false
  esc = false
  text[start..-1].each_char.with_index do |c, i|
    if in_str
      if esc then esc = false
      elsif c == "\\" then esc = true
      elsif c == '"' then in_str = false
      end
    elsif c == '"' then in_str = true
    elsif c == "{" then depth += 1
    elsif c == "}"
      depth -= 1
      if depth.zero?
        return (JSON.parse(text[start, i + 1]) rescue nil)
      end
    end
  end
  nil
end

# Run the grader with a portable hard timeout (pure Ruby — does not depend on a
# GNU `timeout` binary being present).
def grade(model, timeout_s, prompt)
  # Installed plugins' hooks would add routing context (a SessionStart block) the
  # bank never asked for and could rewrite the final answer (a Stop hook); the
  # bootstrap is measured only through --with-bootstrap, so hooks are disabled.
  cmd = ["claude", "--print", "--tools", "", "--settings", '{"disableAllHooks":true}', "--model", model]
  out = +""
  err = +""
  status = nil
  Open3.popen3(*cmd) do |stdin, stdout, stderr, wait_thr|
    # Drain stdout/stderr in threads BEFORE writing so claude can never block on a
    # full output pipe; keep the write+close+wait all under one hard timeout so a
    # stalled or early-exited grader cannot hang or crash the runner.
    #
    # The readers must absorb their own pipe being closed: the timeout path below
    # closes these pipes underneath them on purpose, and a reader that dies of
    # that IOError gets it re-raised by join/value — aborting the whole eval
    # (exit 1, no JSON report) at the exact moment the guard is supposed to
    # report grader_timeout_Ns.
    drain = lambda do |io|
      Thread.new do
        Thread.current.report_on_exception = false
        begin
          io.read
        rescue IOError, SystemCallError
          nil
        end
      end
    end
    out_t = drain.call(stdout)
    err_t = drain.call(stderr)
    begin
      Timeout.timeout(timeout_s) do
        begin
          stdin.write(prompt)
        rescue Errno::EPIPE, IOError
          # claude closed stdin early; its exit status / stderr will explain
        ensure
          stdin.close rescue nil
        end
        status = wait_thr.value
        # Collect reader output INSIDE the timeout: a grader that exits while a
        # child/grandchild still holds the pipe open keeps stdout from reaching
        # EOF, so out_t.value would otherwise hang forever past the guard.
        out = out_t.value.to_s
        err = err_t.value.to_s
      end
    rescue Timeout::Error
      Process.kill("KILL", wait_thr.pid) rescue nil
      stdout.close rescue nil # force the blocked readers to unblock
      stderr.close rescue nil
      out_t.join(1)
      err_t.join(1)
      return [nil, "grader_timeout_#{timeout_s}s"]
    end
  end
  diagnostic = err.strip.empty? ? out.strip : err.strip
  return [nil, "grader_exit_#{status.exitstatus}: #{diagnostic[0, 200]}"] unless status&.success?
  obj = extract_json(out)
  return [nil, "no_json_in_output: #{out.strip[0, 120]}"] unless obj
  [obj, nil]
rescue Errno::ENOENT
  [nil, "claude_not_found"]
end


# --- run ---------------------------------------------------------------------
if dry_run
  puts build_prompt(catalog, tasks.first["utterance"], bootstrap_layer)
  exit 0
end

# Grader availability is decided up front (not via a mid-loop exit code, which a
# `timeout` wrapper would mask as 127). claude missing => advisory skip, exit 0.
if (`command -v claude 2>/dev/null`.strip rescue "").empty?
  puts "eval-routing-bank: grader unavailable (claude CLI not found) — skipped (advisory)"
  exit 0
end

results = []
grader_available = true
tasks.each do |t|
  exp = t["expected_skill"]
  must_not = t["must_not_route_to"] || []
  acceptable = t["acceptable"] || []
  frozen_ref = t["frozen_at_sha"] == "root" ? `git -C #{Shellwords.escape(root)} rev-list --max-parents=0 HEAD`.lines.first.to_s.strip : t["frozen_at_sha"]
  frozen_ok = ancestor?(root, frozen_ref)
  prompt = build_prompt(catalog, t["utterance"], bootstrap_layer)
  verdicts = []
  replicas.times do |ri|
    parsed, error = grade(model, timeout_s, prompt)
    # A verdict that produced no observation contributes nothing to the totals,
    # and an unmeasured task is indistinguishable from a routing failure in the
    # numbers this bank reports. Both recoverable causes are sampling accidents
    # rather than verdicts — a hard timeout is usually machine load, and
    # unparseable output is usually one stray quote in a generated rationale —
    # so each gets one retry.
    # Repairing malformed output instead of re-asking for it was tried and removed:
    # interpreting text that is by definition malformed has no natural boundary,
    # and three review rounds each found a different shape that a repair would read
    # as a verdict. Re-asking needs no such interpretation. Nothing else is retried:
    # an auth failure and a missing CLI do reproduce on a second call.
    if error&.start_with?("grader_timeout_") || error&.start_with?("no_json_in_output")
      parsed, retry_error = grade(model, timeout_s, prompt)
      error = parsed ? nil : retry_error
    end
    if error == "claude_not_found"
      grader_available = false
      break
    end
    selected = parsed && parsed["selected_skill"]
    # A parseable answer is not yet a usable observation. The prompt's contract is
    # an exact catalog name or "none"; anything else -- a missing key, a name the
    # catalog does not carry, a non-string -- is grader output that says nothing
    # about routing, and counting it as a verdict would let it feed the per-case
    # resolution floor exactly as a well-formed one does. It is an ERROR, not a
    # FAIL: a FAIL is evidence against the route, this is absence of evidence.
    if parsed && error.nil? && !(selected == "none" || catalog_names.include?(selected))
      error = "invalid_selection: #{selected.inspect[0, 80]}"
      parsed = nil
    end
    clarify = parsed && parsed["clarify"] == true
    confidence = parsed && parsed["confidence"]
    v_status =
      if error then "ERROR"
      elsif (selected == exp || acceptable.include?(selected)) && !must_not.include?(selected) then "PASS"
      else "FAIL"
      end
    verdicts << { replica: ri + 1, selected: selected, clarify: clarify,
                  confidence: confidence, status: v_status, error: error }
  end
  break unless grader_available
  # Task-level consensus over replicas (replicas=1 reproduces the old per-task
  # semantics exactly). Conservative: one failing replica fails the task —
  # a route that only sometimes lands is not a stable route.
  observed = verdicts.reject { |v| v[:status] == "ERROR" }
  status =
    if observed.empty? then "ERROR"
    elsif observed.all? { |v| v[:status] == "PASS" } then "PASS"
    else "FAIL"
    end
  selections = observed.map { |v| v[:selected] }.uniq
  # Failure-mode labels (vocabulary in references/eval-routing.md):
  #   absorbed        — an utterance that should be rejected (expected "none")
  #                     or kept away from named bait neighbors (must_not_route_to)
  #                     was claimed by such a skill anyway
  #   ownership_split — replicas disagreed on the top pick (unstable ownership)
  labels = []
  absorbed = (exp == "none" && observed.any? { |v| v[:selected] && v[:selected] != "none" }) ||
             observed.any? { |v| must_not.include?(v[:selected]) }
  labels << "absorbed" if absorbed
  labels << "ownership_split" if selections.size > 1
  # A task where some replicas erred but others graded is PARTIALLY measured:
  # the consensus above sees only the observed verdicts, so without this label
  # a PASS+ERROR pair would read as a clean PASS and the run as fully sampled.
  labels << "partial_error" if verdicts.any? { |v| v[:status] == "ERROR" } && !observed.empty?
  results << {
    id: t["id"], utterance: t["utterance"], expected: exp, acceptable: acceptable,
    selected: observed.first && observed.first[:selected],
    confidence: observed.first && observed.first[:confidence],
    clarify: observed.any? { |v| v[:clarify] },
    acceptable_hit: observed.any? { |v| acceptable.include?(v[:selected]) },
    must_not_route_to: must_not, status: status, labels: labels,
    verdicts: verdicts, error: (verdicts.find { |v| v[:error] } || {})[:error],
    frozen_at_sha_is_ancestor: frozen_ok
  }
end

unless grader_available
  puts "eval-routing-bank: grader unavailable (claude CLI not found) — skipped (advisory)"
  exit 0
end

passes = results.count { |r| r[:status] == "PASS" }
fails = results.select { |r| r[:status] == "FAIL" }
errors = results.select { |r| r[:status] == "ERROR" }
drift = results.reject { |r| r[:frozen_at_sha_is_ancestor] }

# First-class routing-quality metrics beyond pass/fail (counted over observed,
# non-ERROR verdicts): clarify rate, low-confidence rate, and — when replicas
# >= 2 — how often all replicas of one task picked the same top skill.
all_observed = results.flat_map { |r| r[:verdicts].reject { |v| v[:status] == "ERROR" } }
clarify_count = all_observed.count { |v| v[:clarify] }
low_conf_count = all_observed.count { |v| v[:confidence].is_a?(Numeric) && v[:confidence] < 0.5 }
# Replica-level error accounting: task-level `error` counts only fully
# unmeasured tasks, so a PASS+ERROR pair would otherwise report zero
# grader-errors while a replica silently went missing.
error_verdicts = results.sum { |r| r[:verdicts].count { |v| v[:status] == "ERROR" } }
partial_error_ids = results.select { |r| r[:labels].include?("partial_error") }.map { |r| r[:id] }
agreement_measured = 0
agreement_agree = 0
if replicas >= 2
  results.each do |r|
    obs = r[:verdicts].reject { |v| v[:status] == "ERROR" }
    next if obs.size < 2
    agreement_measured += 1
    agreement_agree += 1 if obs.map { |v| v[:selected] }.uniq.size == 1
  end
end

# baseline diff (newly failed / newly passed). The baseline is a prior report
# (a hash with a "results" array) or a bare results array. Reports from a
# different bank content or replica count are DIFFERENT RULERS (documented
# above): comparing them emits false regressions/improvements, so a
# demonstrated mismatch suppresses the diff instead of computing it. A bare
# results array carries no fingerprint — its comparability is unverifiable and
# is flagged as such rather than silently trusted.
newly_failed = []
newly_passed = []
baseline_comparable = nil
baseline_incomparable_reason = nil
if baseline_path && File.file?(baseline_path)
  base_json = JSON.parse(File.read(baseline_path))
  if base_json.is_a?(Hash)
    base_bank_sha = base_json.dig("routing_surface", "bank_sha256")
    base_replicas = base_json["replicas"] || 1 # legacy reports predate --replicas
    if base_bank_sha && base_bank_sha != routing_surface[:bank_sha256]
      baseline_comparable = false
      baseline_incomparable_reason = "bank content differs (baseline #{base_bank_sha[0, 12]}… vs current #{routing_surface[:bank_sha256][0, 12]}…)"
    elsif base_replicas != replicas
      baseline_comparable = false
      baseline_incomparable_reason = "replica count differs (baseline #{base_replicas} vs current #{replicas})"
    else
      baseline_comparable = base_bank_sha ? true : "unverified (baseline carries no bank fingerprint)"
    end
  else
    baseline_comparable = "unverified (bare results array carries no fingerprint)"
  end
  unless baseline_comparable == false
    base_results = base_json.is_a?(Hash) ? (base_json["results"] || []) : base_json
    base_status = base_results.to_h { |r| [r["id"], r["status"]] }
    results.each do |r|
      was = base_status[r[:id]]
      newly_failed << r[:id] if was == "PASS" && r[:status] == "FAIL"
      newly_passed << r[:id] if was == "FAIL" && r[:status] == "PASS"
    end
  end
end

# The floor is on VALID observations, not on the requested replica count: a run
# asked for ten replicas can come back with seven usable verdicts once a grader
# times out or returns unparsable output, and a report that called itself
# actionable on the request alone would license an edit the evidence cannot
# support. Observed in this repository: a fourteen-task run at --replicas 10
# returned six grader errors and left two tasks at seven valid observations.
#
# The verdict is PER CASE, because an edit is licensed per case. A report-wide
# flag alone is unsound in the other direction: a subset run over case A can be
# actionable while saying nothing about case B, and a consumer reading only the
# top-level boolean would take it as licence for an edit to B. Each result
# therefore carries its own `actionable`, and the report-level field is the
# conjunction over the cases the run actually measured -- true only when every
# measured case clears the floor, and never a statement about a case absent from
# `results`.
results.each do |r|
  valid = r[:verdicts].count { |v| v[:status] != "ERROR" }
  r[:valid_observations] = valid
  r[:actionable] = replicas >= ACTION_RESOLUTION_MIN_REPLICAS &&
    valid >= ACTION_RESOLUTION_MIN_REPLICAS
end
min_valid_observations = results.map { |r| r[:valid_observations] }.min.to_i
action_resolution = !results.empty? && results.all? { |r| r[:actionable] }

report = {
  model: model, model_source: model_source, tasks: results.size, pass: passes, fail: fails.size, error: errors.size,
  replicas: replicas, verdicts: all_observed.size,
  error_verdicts: error_verdicts, partial_error_tasks: partial_error_ids,
  clarify_count: clarify_count, low_confidence_count: low_conf_count,
  replica_agreement: (replicas >= 2 ? { agree: agreement_agree, measured: agreement_measured } : nil),
  desc_budget_chars: desc_budget, routing_surface: routing_surface,
  action_resolution: action_resolution,
  action_resolution_scope: "cases measured by this run only; see each result's actionable field",
  action_resolution_min_replicas: ACTION_RESOLUTION_MIN_REPLICAS,
  min_valid_observations: min_valid_observations,
  co_change_bank_and_descriptions: co_change, co_change_check_available: co_change_check_ok,
  frozen_drift: drift.map { |r| r[:id] },
  baseline_comparable: baseline_comparable,
  baseline_incomparable_reason: baseline_incomparable_reason,
  newly_failed: newly_failed, newly_passed: newly_passed, results: results
}
File.write(json_path, JSON.pretty_generate(report)) if json_path

puts "eval-routing-bank (#{model}): #{passes}/#{results.size} pass, #{fails.size} fail, #{errors.size} grader-error"
if model_source == "default"
  puts "  router_model_default: #{model} was not chosen with --model; a description edit needs the model tier that routes in use (references/eval-routing.md)"
end
unless action_resolution
  puts "  \u26a0 screening_resolution_only: replicas=#{replicas}, weakest task has #{min_valid_observations} valid observations, floor #{ACTION_RESOLUTION_MIN_REPLICAS} — this report locates candidates, it does not license a description edit; a per-case edit needs #{ACTION_RESOLUTION_MIN_REPLICAS} valid observations of that case (references/eval-routing.md)"
end
puts "  arm: desc-budget-chars=#{desc_budget}" if desc_budget
unless all_observed.empty?
  line = "  clarify: #{clarify_count}/#{all_observed.size} verdicts, low-confidence(<0.5): #{low_conf_count}/#{all_observed.size}"
  line += ", replica top1 agreement: #{agreement_agree}/#{agreement_measured}" if replicas >= 2
  puts line
end
if error_verdicts.positive?
  puts "  ⚠ grader-error verdicts: #{error_verdicts}#{partial_error_ids.empty? ? '' : " (partially measured tasks: #{partial_error_ids.join(', ')})"}"
end
puts "  ⚠ co-change check unavailable: base ref #{base.inspect} not resolvable — could not verify bank/description co-change" unless co_change_check_ok
puts "  ⚠ co-change: this change touches BOTH the bank and a SKILL.md description (verify the bank was not edited to pass)" if co_change
puts "  ⚠ frozen-drift (not ancestor of HEAD, excluded from regression judgment): #{drift.map { |r| r[:id] }.join(', ')}" unless drift.empty?
unless fails.empty?
  puts "  FAIL:"
  fails.each do |r|
    tag = r[:labels].empty? ? "" : " [#{r[:labels].join(',')}]"
    obs = r[:verdicts].reject { |v| v[:status] == "ERROR" }
    got = obs.map { |v| v[:selected].inspect }.uniq.join(" | ")
    conf = obs.map { |v| v[:confidence] }.join(",")
    puts "    - #{r[:id]}: expected #{r[:expected]} got #{got} (conf #{conf})#{tag}"
  end
end
unless errors.empty?
  puts "  GRADER-ERROR:"
  errors.each { |r| puts "    - #{r[:id]}: #{r[:error]}" }
end
if baseline_comparable == false
  puts "  ⚠ baseline not compared — different ruler: #{baseline_incomparable_reason}"
elsif baseline_comparable.is_a?(String)
  puts "  ⚠ baseline comparability #{baseline_comparable}"
end
unless newly_failed.empty? && newly_passed.empty?
  puts "  vs baseline: newly_failed=#{newly_failed.join(',')} newly_passed=#{newly_passed.join(',')}"
end

# Advisory: never block on a routing miss. Only signal grader-infra breakage.
exit 3 if !results.empty? && errors.size == results.size
exit 0
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall