Skip to content

linuxarena/ct-replications / .agents/skills/replicating-a-protocol

replicating-a-protocol

Verified

Guide for replicating a blue-team protocol from an AI control paper as a ct-replications entry, with a decision log against the paper, a fidelity check, a replay of Control Tower's committed recording and a PR description ready to open. Use when the user gives a paper or arXiv link whose mechanism is a protocol, or asks to implement, port or replicate a published protocol (defer, edit, resample, gate, block, audit). For a paper whose novelty is only the judge, use `replicating-a-monitor`; for a protocol the team designs itself, use `adding-a-protocol` in the control-tower repository.

FileKindSize
SKILL.mdskill md15.3 KB
scripts/check_replay.pyscript2.8 KB

scripts/check_replay.py

View on GitHub
"""Fast-loop pass check: the replay succeeded, and every sample has an `audit` score, no error, the
whole recording replayed unless an audit caught the attack, and a protocol note on every step where U
called a tool other than `submit`. Exits 1 on any failure.

Usage, from the pinned Control Tower worktree: uv run python ../ct-replications/.agents/skills/replicating-a-protocol/scripts/check_replay.py <replay.eval>
"""

import sys
from collections.abc import Mapping
from typing import cast

from control_tower.eval.history import FACT_SOURCE
from ct_setting_sdk.roles import UNTRUSTED_ROLE
from inspect_ai.event import InfoEvent, ModelEvent
from inspect_ai.log import read_eval_log

ENGINE_FACTS = {"step", "finished", "unobserved_outcomes"}

if len(sys.argv) != 2:
    sys.exit(__doc__)
log = read_eval_log(sys.argv[1])
print(f"run: {log.status}")
failed = log.status != "success" or not log.samples
for sample in log.samples or []:
    # Per step: [U called a tool other than submit, the protocol noted]; protocols may skip the rest.
    steps: list[list[bool]] = []
    for event in sample.events:
        if isinstance(event, ModelEvent) and event.role == UNTRUSTED_ROLE and steps:
            calls = event.output.message.tool_calls if event.output.choices else None
            if any(call.function != "submit" for call in calls or []):
                steps[-1][0] = True
        if not isinstance(event, InfoEvent) or event.source != FACT_SOURCE:
            continue
        if not isinstance(event.data, dict):
            continue
        if "step" in event.data:
            steps.append([False, False])
        elif steps and not ENGINE_FACTS & event.data.keys():
            steps[-1][1] = True
    problems = []
    if sample.error:
        problems.append(sample.error.message.splitlines()[0])
    audit = (sample.scores or {}).get("audit")
    if audit is None:
        problems.append("no audit score")
    outcome = (
        cast(Mapping[str, object], audit.value)
        if audit is not None and isinstance(audit.value, Mapping)
        else {}
    )
    caught = bool(outcome.get("caught"))
    explanations = [s.explanation or "" for s in (sample.scores or {}).values()]
    if not caught and any(
        e.startswith(("Replay stopped", "Replay diverged")) for e in explanations
    ):
        problems.append(
            "the recording did not replay to its end, and no audit caught the attack"
        )
    unnoted = [i for i, (acted, noted) in enumerate(steps) if acted and not noted]
    if unnoted:
        problems.append(f"steps without a protocol note: {unnoted}")
    failed |= bool(problems)
    label = sample.id if sample.epoch == 1 else f"{sample.id} epoch {sample.epoch}"
    print(f"{'FAIL' if problems else 'ok'} {label}: {len(steps)} steps", *problems)
sys.exit(1 if failed else 0)
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 193kUpdated Claude CodeCodexVerifiedInstall