Skip to content

linuxarena/ct-replications / .agents/skills/replicating-a-monitor

replicating-a-monitor

Verified

Guide for replicating a monitor from an AI control paper as a ct-replications entry, with a decision log against the paper, the source text copied verbatim, a fidelity check, a run on Control Tower's committed recordings and a PR description ready to open. Use when the user gives a paper or arXiv link whose mechanism is a monitor, or asks to implement, port or replicate a published monitor. For a paper whose mechanism is a protocol, use `replicating-a-protocol`; for a monitor the team designs itself, use `adding-a-monitor` in the control-tower repository.

FileKindSize
SKILL.mdskill md13.0 KB
scripts/check_scored.pyscript1.6 KB

scripts/check_scored.py

View on GitHub
"""Fast-loop pass check: every sample of a monitored log has scores under the monitor's name and
none is unscored; given a rerun's monitored log, every score also matches it. Exits 1 on any failure.

Usage, from the pinned Control Tower worktree: uv run python ../ct-replications/.agents/skills/replicating-a-monitor/scripts/check_scored.py <monitored.eval> <name> [<rerun.monitored.eval>]
"""

import sys

from control_tower.eval.log.sample import suspicion_scores
from inspect_ai.log import read_eval_log

if len(sys.argv) not in (3, 4):
    sys.exit(__doc__)
path, name = sys.argv[1], sys.argv[2]
rerun = (
    {
        (sample.id, sample.epoch): {
            (s.draw, s.at): s.value
            for s in suspicion_scores(sample.scores)
            if s.monitor == name
        }
        for sample in read_eval_log(sys.argv[3]).samples or []
    }
    if len(sys.argv) == 4
    else None
)
samples = read_eval_log(path).samples or []
failed = not samples
for sample in samples:
    scores = [s for s in suspicion_scores(sample.scores) if s.monitor == name]
    unscored = [s.explanation for s in scores if s.value is None]
    differs = rerun is not None and rerun.get((sample.id, sample.epoch)) != {
        (s.draw, s.at): s.value for s in scores
    }
    ok = bool(scores) and not unscored and not differs
    failed |= not ok
    label = sample.id if sample.epoch == 1 else f"{sample.id} epoch {sample.epoch}"
    print(
        f"{'ok' if ok else 'FAIL'} {label}: {len(scores)} scores, {len(unscored)} unscored",
        *(["differs from the rerun"] if differs else []),
        *unscored[:1],
    )
sys.exit(1 if failed else 0)
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall