me2resh/apexyard / .claude/skills/eval-agents
eval-agents
UnverifiedScore a review agent (Rex/Hakim/Tariq) against a labeled PR corpus — ground-truth defect-set overlap, never a prose rubric. Headline metric is approve-precision.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 28.8 KB |
| lib/mine_disagreements.pyexecutable | script | 23.1 KB |
| lib/snapshot-diff.shexecutable | script | 3.3 KB |
| lib/source-snapshot.shexecutable | script | 6.7 KB |
| lib/validate-corpus.shexecutable | script | 8.7 KB |
| tests/fixtures/mine-disagreements/edge-cases.json | config | 4.9 KB |
| tests/fixtures/mine-disagreements/merged-prs.json | config | 10.8 KB |
| tests/smoke.shexecutable | script | 3.4 KB |
| tests/test-snapshot-diff.shexecutable | script | 3.2 KB |
| tests/test_mine_disagreements.py | script | 17.0 KB |
| tests/test_mine_disagreements.shexecutable | script | 5.6 KB |
| tests/test_source_snapshot.shexecutable | script | 11.8 KB |
lib/snapshot-diff.sh
View on GitHub#!/usr/bin/env bash
# snapshot-diff.sh — filesystem contamination detector for /eval-agents.
#
# Snapshots a directory's file set before an agent-under-test spawn, then
# checks it after. Any new, changed, or deleted file is flagged. This is the
# authoritative contamination check (SKILL.md Step 3c/3e, #833) — a stronger
# backstop than scanning the agent's returned text for a marker-write
# confession, because a silent Bash-redirect write never has to appear in
# narrated output for this check to catch it.
#
# Usage:
# snapshot-diff.sh snapshot <dir> <snapshot-file>
# snapshot-diff.sh check <dir> <snapshot-file>
#
# `snapshot` records a manifest of <dir>'s current file set (relative path +
# sha256, one per line, sorted) to <snapshot-file>. `check` re-scans <dir>
# and reports any file that's NEW, CHANGED (different hash), or DELETED
# since the snapshot was taken.
#
# Exit codes:
# 0 — no contamination (dir unchanged since snapshot)
# 1 — contamination detected (each affected path + reason printed to stdout)
# 2 — usage error (bad args, or `check` with no prior `snapshot`)
set -euo pipefail
export LC_ALL=C # deterministic sort/comm collation regardless of locale
MODE="${1:-}"
DIR="${2:-}"
SNAPSHOT_FILE="${3:-}"
usage() {
echo "usage: snapshot-diff.sh <snapshot|check> <dir> <snapshot-file>" >&2
exit 2
}
[ -n "$MODE" ] && [ -n "$DIR" ] && [ -n "$SNAPSHOT_FILE" ] || usage
sha_of() {
# Portable sha256: shasum on macOS, sha256sum on Linux.
if command -v shasum >/dev/null 2>&1; then
shasum -a 256 "$1" | awk '{print $1}'
else
sha256sum "$1" | awk '{print $1}'
fi
}
manifest_of() {
# Emits "<relpath> <sha256>" for every regular file under $DIR, sorted.
# Creates $DIR if missing — a fresh session may have no reviews/ dir yet,
# and "the directory doesn't exist" is not itself contamination.
mkdir -p "$DIR"
find "$DIR" -type f 2>/dev/null | sort | while IFS= read -r f; do
rel="${f#"$DIR"/}"
echo "$rel $(sha_of "$f")"
done
}
case "$MODE" in
snapshot)
manifest_of > "$SNAPSHOT_FILE"
;;
check)
if [ ! -f "$SNAPSHOT_FILE" ]; then
echo "snapshot-diff.sh: no snapshot at $SNAPSHOT_FILE — run 'snapshot' first" >&2
exit 2
fi
CURRENT="$(mktemp)"
trap 'rm -f "$CURRENT"' EXIT
manifest_of > "$CURRENT"
if diff -q "$SNAPSHOT_FILE" "$CURRENT" >/dev/null 2>&1; then
echo "OK — no changes to $DIR since snapshot"
exit 0
fi
echo "CONTAMINATION — $DIR changed since snapshot:"
path_in() {
# $1 = path, $2 = manifest file — fixed-string match on the "path " prefix
grep -qF "$1 " "$2" 2>/dev/null
}
# Lines only in CURRENT (new content, or a path whose hash changed).
comm -13 "$SNAPSHOT_FILE" "$CURRENT" | while IFS= read -r line; do
path="${line%% *}"
if path_in "$path" "$SNAPSHOT_FILE"; then
echo " CHANGED: $path"
else
echo " NEW: $path"
fi
done
# Lines only in SNAPSHOT and whose path is genuinely gone from CURRENT
# (not just hash-changed — that case was already reported as CHANGED above).
comm -23 "$SNAPSHOT_FILE" "$CURRENT" | while IFS= read -r line; do
path="${line%% *}"
if ! path_in "$path" "$CURRENT"; then
echo " DELETED: $path"
fi
done
exit 1
;;
*)
usage
;;
esac
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |