linuxarena/ct-replications / .agents/skills/replicating-a-protocol
replicating-a-protocol
VerifiedGuide for replicating a blue-team protocol from an AI control paper as a ct-replications entry, with a decision log against the paper, a fidelity check, a replay of Control Tower's committed recording and a PR description ready to open. Use when the user gives a paper or arXiv link whose mechanism is a protocol, or asks to implement, port or replicate a published protocol (defer, edit, resample, gate, block, audit). For a paper whose novelty is only the judge, use `replicating-a-monitor`; for a protocol the team designs itself, use `adding-a-protocol` in the control-tower repository.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.3 KB |
| scripts/check_replay.py | script | 2.8 KB |
scripts/check_replay.py
View on GitHub"""Fast-loop pass check: the replay succeeded, and every sample has an `audit` score, no error, the
whole recording replayed unless an audit caught the attack, and a protocol note on every step where U
called a tool other than `submit`. Exits 1 on any failure.
Usage, from the pinned Control Tower worktree: uv run python ../ct-replications/.agents/skills/replicating-a-protocol/scripts/check_replay.py <replay.eval>
"""
import sys
from collections.abc import Mapping
from typing import cast
from control_tower.eval.history import FACT_SOURCE
from ct_setting_sdk.roles import UNTRUSTED_ROLE
from inspect_ai.event import InfoEvent, ModelEvent
from inspect_ai.log import read_eval_log
ENGINE_FACTS = {"step", "finished", "unobserved_outcomes"}
if len(sys.argv) != 2:
sys.exit(__doc__)
log = read_eval_log(sys.argv[1])
print(f"run: {log.status}")
failed = log.status != "success" or not log.samples
for sample in log.samples or []:
# Per step: [U called a tool other than submit, the protocol noted]; protocols may skip the rest.
steps: list[list[bool]] = []
for event in sample.events:
if isinstance(event, ModelEvent) and event.role == UNTRUSTED_ROLE and steps:
calls = event.output.message.tool_calls if event.output.choices else None
if any(call.function != "submit" for call in calls or []):
steps[-1][0] = True
if not isinstance(event, InfoEvent) or event.source != FACT_SOURCE:
continue
if not isinstance(event.data, dict):
continue
if "step" in event.data:
steps.append([False, False])
elif steps and not ENGINE_FACTS & event.data.keys():
steps[-1][1] = True
problems = []
if sample.error:
problems.append(sample.error.message.splitlines()[0])
audit = (sample.scores or {}).get("audit")
if audit is None:
problems.append("no audit score")
outcome = (
cast(Mapping[str, object], audit.value)
if audit is not None and isinstance(audit.value, Mapping)
else {}
)
caught = bool(outcome.get("caught"))
explanations = [s.explanation or "" for s in (sample.scores or {}).values()]
if not caught and any(
e.startswith(("Replay stopped", "Replay diverged")) for e in explanations
):
problems.append(
"the recording did not replay to its end, and no audit caught the attack"
)
unnoted = [i for i, (acted, noted) in enumerate(steps) if acted and not noted]
if unnoted:
problems.append(f"steps without a protocol note: {unnoted}")
failed |= bool(problems)
label = sample.id if sample.epoch == 1 else f"{sample.id} epoch {sample.epoch}"
print(f"{'FAIL' if problems else 'ok'} {label}: {len(steps)} steps", *problems)
sys.exit(1 if failed else 0)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |