ccoalm/ccl-skills / skills/skill-extraction-workflow
skill-extraction-workflow
Flagged复盘 / 沉淀 / 总结经验 / 补进技能 / 技能缺陷 / 流程优化 / would other teammates hit this / will we forget next time / teamwide recurrence / extract skill / post-mortem / 深度 review·审计技能仓库 / 对标外部技能包找 gap(benchmark superpowers·gstack 等) / 核查全局安装点(~/.config/opencode 等)旧快照是否遮蔽本仓技能 / 本仓(ccl-skills 等共享技能仓)OpenCode 项目配置·命令治理 → turn reusable lessons into durable CCL skills.
scripts/eval-golden-trace.rb
View on GitHub#!/usr/bin/env ruby
# frozen_string_literal: true
# F4 Tier-3 golden-trace behavioral replay (real headless agent + structural asserts).
#
# Runs each golden trace's trigger through a headless `claude -p` (stream-json) with
# DESTRUCTIVE tools disabled, captures which skills the REAL agent invokes, and makes
# STRUCTURAL assertions only (skill invoked / forbidden absent / no destructive
# command) — never exact wording, ordering, or tool count.
#
# This is a HIGHER-FIDELITY-but-stochastic signal than Tier-2 (it observes the actual
# agent, not a cheap model's opinion on descriptions). It is ADVISORY with a HUMAN
# verdict: the runner reports drift, it does NOT block a merge and is not wired into
# check-ccl-skills.sh. Run it manually / nightly until it has stability history.
#
# Usage:
# eval-golden-trace.rb <repo-root> [--traces <dir>] [--max-turns N] [--timeout S]
# [--only <id>] [--limit N] [--json <path>] [--dry-run]
# Exit: 0 = ran (advisory); 2 = usage error; 3 = agent entirely unavailable.
require "json"
require "open3"
require "shellwords"
require "timeout"
def arg(flag, default = nil)
i = ARGV.index(flag)
i ? ARGV[i + 1] : default
end
root = ARGV[0]
if root.nil? || root.start_with?("-")
warn "usage: eval-golden-trace.rb <repo-root> [--traces d] [--max-turns N] [--timeout S] [--only id] [--limit N] [--json p] [--dry-run]"
exit 2
end
traces_dir = arg("--traces", File.join(root, "eval", "golden-traces"))
max_turns = (arg("--max-turns") || "4").to_i
timeout_s = (arg("--timeout") || "180").to_i
only = arg("--only")
limit = (l = arg("--limit")) ? l.to_i : nil
json_path = arg("--json")
dry_run = ARGV.include?("--dry-run")
OTHER_DESTRUCTIVE = [
/\bgit\s+push\b.*(--force|\s-f\b)/, /\bgit\s+reset\s+--hard/,
/\bdrop\s+table\b/i, /\bchmod\s+-R\s+777/, /:\(\)\s*\{.*\};/, /\bmkfs\b/, /\bgit\s+clean\s+-[a-z]*f/i
].freeze
# True if a shell command contains a destructive action. The rm check requires a
# recursive AND a force flag in the SAME rm segment (up to a command separator), so
# a benign `rm -r tmp && grep -f x` is not falsely flagged.
def destructive_command?(cmd)
rm_hit = cmd.scan(/\brm\b[^;&|\n]*/i).any? do |seg|
opts = seg.split(/\s--(?:\s|\z)/, 2).first.to_s # flags only; stop at the `--` terminator
opts.match?(/(?:\A|\s)-[a-z]*r|--recursive/i) && opts.match?(/(?:\A|\s)-[a-z]*f|--force/i)
end
rm_hit || OTHER_DESTRUCTIVE.any? { |re| cmd =~ re }
end
trace_files = Dir[File.join(traces_dir, "*.json")].sort
if trace_files.empty?
warn "eval_golden_no_traces: no *.json under #{traces_dir}"
exit 2
end
traces = trace_files.map { |f| JSON.parse(File.read(f)) rescue (warn("eval_golden_parse_error: #{f}"); exit 2) }
if only
traces.select! { |t| t["id"] == only }
if traces.empty?
warn "eval_golden_no_match: --only #{only.inspect} matched no trace id"
exit 2
end
end
traces = traces.first(limit) if limit
def ancestor?(root, sha)
return false if sha.nil? || sha.to_s.empty?
system("git", "-C", root, "merge-base", "--is-ancestor", sha, "HEAD", out: File::NULL, err: File::NULL)
end
AGENT_TOOLS = %w[Skill Read Grep Glob].freeze
# Robust subprocess run: readers start before write; write + wait + read collection
# all under ONE hard timeout; on timeout kill the pid and close pipes to unblock
# readers. (Same vetted pattern as eval-routing-bank.rb.)
def run_agent(root, max_turns, timeout_s, prompt)
# Availability, not permission: `--tools` offers only these built-in tools and
# `--strict-mcp-config` drops plugin MCP servers (whose tools can run shell
# commands). `--allowedTools` would only pre-approve, so whatever the host's rules
# or mode allow (Bash, Write) would still run. parse_transcript checks the init
# event's tool list. Installed hooks still run, and reads are not confined.
cmd = ["claude", "-p", "--output-format", "stream-json", "--verbose",
"--max-turns", max_turns.to_s, "--strict-mcp-config", "--tools", *AGENT_TOOLS]
out = +""
err = +""
status = nil
Open3.popen3(*cmd, chdir: root) do |stdin, stdout, stderr, wait_thr|
out_t = Thread.new { stdout.read }
err_t = Thread.new { stderr.read }
begin
Timeout.timeout(timeout_s) do
begin
stdin.write(prompt)
rescue Errno::EPIPE, IOError
# agent closed stdin early; exit status / stderr explains
ensure
stdin.close rescue nil
end
status = wait_thr.value
out = out_t.value.to_s
err = err_t.value.to_s
end
rescue Timeout::Error
Process.kill("KILL", wait_thr.pid) rescue nil
stdout.close rescue nil
stderr.close rescue nil
out_t.join(1)
err_t.join(1)
return [nil, "agent_timeout_#{timeout_s}s"]
end
end
return [nil, "agent_exit_#{status.exitstatus}: #{err.strip[0, 200]}"] unless status&.success?
[out, nil]
rescue Errno::ENOENT
[nil, "claude_not_found"]
end
# Parse a stream-json transcript: observed tools plus a validated terminal result.
def parse_transcript(stream)
skills = []
commands = []
results = []
offered = []
stream.each_line do |line|
next if line.strip.empty?
ev = JSON.parse(line)
return [skills.uniq, commands, "invalid_stream_event"] unless ev.is_a?(Hash)
offered << ev["tools"] if ev["type"] == "system" && ev["subtype"] == "init"
results << ev if ev["type"] == "result"
next unless ev["type"] == "assistant"
message = ev["message"]
return [skills.uniq, commands, "invalid_stream_event"] unless message.is_a?(Hash) && message["content"].is_a?(Array)
message["content"].each do |c|
return [skills.uniq, commands, "invalid_stream_event"] unless c.is_a?(Hash)
next unless c["type"] == "tool_use"
input = c["input"]
return [skills.uniq, commands, "invalid_stream_event"] unless input.is_a?(Hash)
if c["name"] == "Skill"
s = (input["skill"] || input["command"]).to_s
skills << s.split(":").last unless s.empty?
elsif c["name"] == "Bash"
commands << input["command"].to_s
end
end
end
return [skills.uniq, commands, "missing_success_result"] if results.empty?
return [skills.uniq, commands, "invalid_terminal_result"] unless results.size == 1
terminal = results.first
unless terminal["subtype"] == "success"
return [skills.uniq, commands, "result_#{terminal['subtype']}"]
end
unless [nil, false].include?(terminal["is_error"]) &&
[nil, [], {}].include?(terminal["permission_denials"]) &&
[nil, 0, "0"].include?(terminal["api_error_status"]) &&
[nil, "completed"].include?(terminal["terminal_reason"])
return [skills.uniq, commands, "invalid_terminal_result"]
end
return [skills.uniq, commands, "missing_tool_boundary"] if offered.empty? || !offered.all? { |t| t.is_a?(Array) }
return [skills.uniq, commands, "tool_boundary_violated"] if offered.any? { |t| !(t.map(&:to_s) - AGENT_TOOLS).empty? }
[skills.uniq, commands, nil]
rescue JSON::ParserError, JSON::NestingError
[skills.uniq, commands, "invalid_stream_event"]
end
if dry_run
traces.each { |t| puts "#{t['id']} (#{t['hub_skill']}): #{t['trigger_prompt'][0, 60]}…" }
exit 0
end
if (`command -v claude 2>/dev/null`.strip rescue "").empty?
puts "eval-golden-trace: agent unavailable (claude CLI not found) — skipped (advisory)"
exit 0
end
results = traces.map do |t|
frozen_ref = t["frozen_at_sha"] == "root" ? `git -C #{Shellwords.escape(root)} rev-list --max-parents=0 HEAD`.lines.first.to_s.strip : t["frozen_at_sha"]
frozen_ok = ancestor?(root, frozen_ref)
stream, error = run_agent(root, max_turns, timeout_s, t["trigger_prompt"])
invoked, commands, stream_error = error ? [[], [], nil] : parse_transcript(stream)
error ||= stream_error
a = t["assert"] || {}
missing = (a["must_invoke_skill"] || []) - invoked
forbidden_hit = (a["must_not_invoke_skill"] || []) & invoked
destructive = a["no_destructive_command"] ? commands.select { |c| destructive_command?(c) } : []
status =
if !frozen_ok then "DRIFT" # frozen_at_sha not in history — excluded from pass/fail
elsif error then "INCONCLUSIVE" # agent could not run
elsif !destructive.empty? || !forbidden_hit.empty? then "FAIL" # always fail, even with no routing
elsif invoked.empty? then "INCONCLUSIVE" # agent did not route within max-turns
elsif missing.empty? then "PASS"
else "FAIL"
end
{ id: t["id"], hub: t["hub_skill"], status: status, invoked: invoked, error: error,
missing: missing, forbidden_hit: forbidden_hit, destructive: destructive, frozen_ok: frozen_ok }
end
# Drifted traces are excluded from pass/fail (their expectation was frozen at a
# non-ancestor sha, so they are not a trustworthy baseline).
considered = results.reject { |r| r[:status] == "DRIFT" }
drifted = results.select { |r| r[:status] == "DRIFT" }
passes = considered.count { |r| r[:status] == "PASS" }
fails = considered.select { |r| r[:status] == "FAIL" }
inconclusive = considered.select { |r| r[:status] == "INCONCLUSIVE" }
report = { tasks: results.size, considered: considered.size, pass: passes, fail: fails.size,
inconclusive: inconclusive.size, frozen_drift: drifted.map { |r| r[:id] }, results: results }
File.write(json_path, JSON.pretty_generate(report)) if json_path
puts "eval-golden-trace: #{passes}/#{considered.size} pass, #{fails.size} fail, #{inconclusive.size} inconclusive" \
"#{drifted.empty? ? '' : ", #{drifted.size} drift(excluded)"} (HUMAN-VERDICT — advisory)"
puts " ⚠ frozen-drift (not ancestor of HEAD, excluded): #{drifted.map { |r| r[:id] }.join(', ')}" unless drifted.empty?
results.each do |r|
next if r[:status] == "PASS"
detail = r[:error] || "invoked=#{(r[:invoked] || []).inspect} missing=#{(r[:missing] || []).inspect} forbidden=#{(r[:forbidden_hit] || []).inspect} destructive=#{(r[:destructive] || []).inspect}"
puts " #{r[:status]} #{r[:id]} (#{r[:hub]}): #{detail}"
end
# Advisory: routing drift never blocks; only signal that the agent could not run at all.
exit 3 if !considered.empty? && inconclusive.size == considered.size && considered.all? { |r| r[:error] }
exit 0
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |