linuxarena/ct-replications / .agents/skills/replicating-a-monitor
replicating-a-monitor
VerifiedGuide for replicating a monitor from an AI control paper as a ct-replications entry, with a decision log against the paper, the source text copied verbatim, a fidelity check, a run on Control Tower's committed recordings and a PR description ready to open. Use when the user gives a paper or arXiv link whose mechanism is a monitor, or asks to implement, port or replicate a published monitor. For a paper whose mechanism is a protocol, use `replicating-a-protocol`; for a monitor the team designs itself, use `adding-a-monitor` in the control-tower repository.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.0 KB |
| scripts/check_scored.py | script | 1.6 KB |
scripts/check_scored.py
View on GitHub"""Fast-loop pass check: every sample of a monitored log has scores under the monitor's name and
none is unscored; given a rerun's monitored log, every score also matches it. Exits 1 on any failure.
Usage, from the pinned Control Tower worktree: uv run python ../ct-replications/.agents/skills/replicating-a-monitor/scripts/check_scored.py <monitored.eval> <name> [<rerun.monitored.eval>]
"""
import sys
from control_tower.eval.log.sample import suspicion_scores
from inspect_ai.log import read_eval_log
if len(sys.argv) not in (3, 4):
sys.exit(__doc__)
path, name = sys.argv[1], sys.argv[2]
rerun = (
{
(sample.id, sample.epoch): {
(s.draw, s.at): s.value
for s in suspicion_scores(sample.scores)
if s.monitor == name
}
for sample in read_eval_log(sys.argv[3]).samples or []
}
if len(sys.argv) == 4
else None
)
samples = read_eval_log(path).samples or []
failed = not samples
for sample in samples:
scores = [s for s in suspicion_scores(sample.scores) if s.monitor == name]
unscored = [s.explanation for s in scores if s.value is None]
differs = rerun is not None and rerun.get((sample.id, sample.epoch)) != {
(s.draw, s.at): s.value for s in scores
}
ok = bool(scores) and not unscored and not differs
failed |= not ok
label = sample.id if sample.epoch == 1 else f"{sample.id} epoch {sample.epoch}"
print(
f"{'ok' if ok else 'FAIL'} {label}: {len(scores)} scores, {len(unscored)} unscored",
*(["differs from the rerun"] if differs else []),
*unscored[:1],
)
sys.exit(1 if failed else 0)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |