Skip to content

anyshift-io/sre-skills / skills/iam-deceptive-escalation-auditor

iam-deceptive-escalation-auditor

Verified

FileKindSize
FAILURE_MODES.mdmarkdown3.6 KB
SKILL.mdskill md19.6 KB
fixtures/01-orphaned-passrole-deny/meta.jsonconfig349 B
fixtures/01-orphaned-passrole-deny/policy.jsonconfig573 B
fixtures/02-action-star-blanket-deny/meta.jsonconfig408 B
fixtures/02-action-star-blanket-deny/policy.jsonconfig604 B
fixtures/03-assumerole-broken-trust/meta.jsonconfig492 B
fixtures/03-assumerole-broken-trust/policy.jsonconfig492 B
fixtures/03-assumerole-broken-trust/trust-policy.jsonconfig473 B
fixtures/05-iam-mutation-boundary-capped/meta.jsonconfig696 B
fixtures/05-iam-mutation-boundary-capped/policy-1.jsonconfig550 B
fixtures/05-iam-mutation-boundary-capped/policy-2.jsonconfig945 B
fixtures/06-cross-account-assume-condition-gated/meta.jsonconfig700 B
fixtures/06-cross-account-assume-condition-gated/policy-1.jsonconfig690 B
fixtures/06-cross-account-assume-condition-gated/policy-2.jsonconfig512 B
fixtures/06-cross-account-assume-condition-gated/trust-policy.jsonconfig400 B
fixtures/07-passrole-sandboxed-role-orphaned/meta.jsonconfig997 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-1.jsonconfig508 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-2.jsonconfig571 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-3.jsonconfig640 B
fixtures/08-ml-platform-passrole-launch-needle/meta.jsonconfig1.0 KB
fixtures/08-ml-platform-passrole-launch-needle/policy-1.jsonconfig713 B
fixtures/08-ml-platform-passrole-launch-needle/policy-2.jsonconfig670 B
fixtures/08-ml-platform-passrole-launch-needle/policy-3.jsonconfig745 B
fixtures/08-ml-platform-passrole-launch-needle/policy-4.jsonconfig647 B
fixtures/08-ml-platform-passrole-launch-needle/policy-5.jsonconfig959 B
fixtures/08-ml-platform-passrole-launch-needle/policy-6.jsonconfig568 B
tests/README.mdmarkdown3.8 KB
tests/_audit.pyscript32.5 KB
tests/_replay.pyscript720 B
tests/eval/README.mdmarkdown5.5 KB
tests/eval/eval_results.jsonconfig493.4 KB
tests/eval/judge_prompt.mdmarkdown3.0 KB
tests/eval/rubric.mdmarkdown3.3 KB
tests/eval/run_eval.pyscript16.1 KB
tests/eval/scenarios.pyscript11.2 KB
tests/replay_01_orphaned_passrole_deny.pyscript1.5 KB
tests/replay_02_action_star_blanket_deny.pyscript1.4 KB
tests/replay_03_assumerole_broken_trust.pyscript1.5 KB
tests/replay_05_iam_mutation_boundary_capped.pyscript2.1 KB
tests/replay_06_cross_account_assume_condition_gated.pyscript1.8 KB
tests/replay_07_passrole_sandboxed_role_orphaned.pyscript2.0 KB
tests/replay_08_ml_platform_passrole_launch_needle.pyscript2.3 KB

tests/eval/run_eval.py

View on GitHub
"""
Control-only screening eval for the candidate iam-deceptive-escalation-auditor skill.

There is NO SKILL.md and NO treatment arm here. This harness measures one thing: does a
COLD agent (no skill, generic "review this policy for problems" prompt) fail on this
domain? A skill is worth building only if the cold agent scores LOW (mean < 4/7) on these
deliberately deceptive fixtures.

For each fixture, runs N trials in the control condition only:
- Control: the agent is given the raw IAM policy JSON and a generic "audit this policy"
  prompt. It uses whatever it brings from training.

Each agent output is graded against the 7-item rubric (rubric.md) by an LLM judge, anchored
to the deterministic reference audit (_audit.py, via scenarios.py) as ground truth.

Final report: per-fixture mean control score, stdev across trials, the aggregate mean, and
a screening verdict (build the skill only if the cold agent is weak here).

Requirements:
- ANTHROPIC_API_KEY environment variable.
- `pip install anthropic` (the only non-stdlib dependency in the repo; isolated to tests/eval/).

Usage:
    python tests/eval/run_eval.py --trials 5
    python tests/eval/run_eval.py --trials 1 --fixtures 02,05,07   # smoke test
    python tests/eval/run_eval.py --trials 5 --fresh               # ignore prior results

Resume: each completed trial is persisted immediately (atomic temp+rename), and a re-run
reloads what is already on disk and fills only the missing (fixture, trial) cells. A crash,
an interrupt, or an API overload mid-run therefore never throws away completed work -- just
re-run the same command to finish. Pass --fresh to start clean.

Cost note: 7 fixtures x 1 condition x 5 trials, plus a judge call per output, is
~35 agent calls + ~35 judge calls = ~70 LLM calls. Expect a few dollars depending on model.
"""

from __future__ import annotations

import argparse
import json
import os
import statistics
import sys
import time
from pathlib import Path

try:
    from anthropic import Anthropic
except ImportError:
    print("ERROR: anthropic SDK not installed. Run: pip install anthropic", file=sys.stderr)
    sys.exit(1)

sys.path.insert(0, str(Path(__file__).parent))
from scenarios import SCENARIOS, load_fixture_text, expected_audit  # noqa: E402

DEFAULT_AGENT_MODEL = os.environ.get("EVAL_AGENT_MODEL", "claude-sonnet-4-6")
DEFAULT_JUDGE_MODEL = os.environ.get("EVAL_JUDGE_MODEL", "claude-sonnet-4-6")
MAX_TOKENS = 4096

# SKILL.md lives at the skill root (three levels up from tests/eval/).
SKILL_MD = Path(__file__).resolve().parents[2] / "SKILL.md"


def build_control_prompt(scenario: dict) -> str:
    return f"""You are a cloud security engineer reviewing an AWS IAM policy for misconfigurations.

Principal: {scenario['principal']}
Context: {scenario['role']}

Here is the policy, exactly as returned by the IAM API:

{load_fixture_text(scenario)}

Audit this policy. Identify any over-broad grant or privilege-escalation path: anything that
lets this principal do more than it should, or escalate to administrator. For each, give the
severity and the fix. Be specific."""


def build_treatment_prompt(scenario: dict, skill_md_text: str) -> str:
    """Treatment arm: the SAME audit ask and the SAME policy JSON as control, plus SKILL.md
    prepended as the methodology to apply. The only variable between arms is the skill, so
    the lift is attributable to it."""
    return f"""You are a cloud security engineer reviewing an AWS IAM policy for misconfigurations. Apply the methodology below to the policy that follows it.

==== METHODOLOGY (SKILL.md) ====
{skill_md_text}
==== END METHODOLOGY ====

Principal: {scenario['principal']}
Context: {scenario['role']}

Here is the policy, exactly as returned by the IAM API:

{load_fixture_text(scenario)}

Apply the methodology above. Identify any over-broad grant or privilege-escalation path, or
state that an apparent escalation is neutralised. For each finding give the severity and the
fix, and name what you cannot determine from the policy alone. Be specific."""


JUDGE_SYSTEM = """You are an expert AWS / cloud-security evaluator grading an IAM policy audit against a 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial credit.

You will be given a known-good answer from a deterministic reference audit, the agent's audit output, and the 7 rubric items.

Return JSON only (no prose), with this exact schema:

{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence>", ...]
}"""


def build_judge_prompt(scenario: dict, agent_output: str) -> str:
    exp = expected_audit(scenario)
    return f"""FIXTURE: {scenario['id']}
PRINCIPAL: {scenario['principal']}

KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): {exp['codes']}  (severity: {exp['top_severity']}; clean: {exp['clean']})
- What the defect is: {scenario['expected_headline']}
- Correct top fix: {scenario['expected_top_fix']}
- Boundary the audit cannot cross: {scenario['expected_boundary_join']}

AGENT AUDIT OUTPUT:
{agent_output}

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: reads the statements correctly, including expanding wildcard Actions (a `*` or `svc:*` is recognised as the concrete sensitive permissions it grants, not skipped).
2. Findings: identifies the ground-truth defect(s) by substance (the same escalation path or over-broad grant; it need not use the rule code), or correctly reports none for a deceptive-clean fixture.
3. No false positives: does not assert a material defect the ground truth lacks (esp. not calling a Deny-neutralised PassRole, a Resource-scoped Action '*', a broken-back AssumeRole, or a read-only wildcard a live privilege escalation).
4. Cross-statement reasoning: where the defect is a combination (PassRole + a launch action split across attached policies, etc.), the agent evaluates the union of statements rather than clearing each statement in isolation; and conversely recognises when a Deny or a scope neutralises an apparent combo.
5. Criticality: ranks a real escalation-to-admin as most serious, and does NOT headline a neutralised or read-only grant as critical.
6. Boundary: names at least one thing it cannot determine from the policy alone, matching the ground-truth join (other attached policies, permissions boundary, org SCPs, the target role's privileges).
7. Recommendation: top fix matches the ground-truth fix in substance (scope the action/resource, remove the escalation grant, add a condition; or correctly state no fix is needed for a neutralised policy).

Return JSON only."""


RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 529}
MAX_RETRIES = 6


def _with_retries(fn, *args, **kwargs):
    """Call fn with exponential backoff on transient API errors (429/5xx/529/overloaded).

    The Anthropic SDK already retries a couple of times; this widens the window so a
    multi-minute overload spell drops far fewer trials. Re-raises on non-retryable
    errors or once retries are exhausted.

    Returns (result, call_seconds) where call_seconds is the wall-time of the SUCCESSFUL
    attempt only -- backoff sleeps and failed attempts are excluded, so duration metrics
    reflect real audit latency, not how overloaded the API happened to be.
    """
    delay = 2.0
    last_exc = None
    for attempt in range(MAX_RETRIES):
        try:
            t_call = time.time()
            return fn(*args, **kwargs), time.time() - t_call
        except Exception as e:  # noqa: BLE001 - inspect, then decide retryable
            status = getattr(e, "status_code", None)
            msg = str(e).lower()
            retryable = status in RETRYABLE_STATUS or "overloaded" in msg or "rate" in msg or "timeout" in msg
            if not retryable:
                raise
            last_exc = e
            if attempt < MAX_RETRIES - 1:
                time.sleep(delay)
                delay = min(delay * 2, 60.0)
    raise last_exc


def run_agent(client: Anthropic, model: str, prompt: str) -> tuple[str, float]:
    """Returns (agent_output_text, audit_seconds). Seconds excludes retry backoff."""
    def _call():
        return client.messages.create(
            model=model,
            max_tokens=MAX_TOKENS,
            messages=[{"role": "user", "content": prompt}],
        )
    resp, call_s = _with_retries(_call)
    return "".join(block.text for block in resp.content if block.type == "text"), call_s


def run_judge(client: Anthropic, model: str, scenario: dict, agent_output: str) -> dict:
    def _call():
        return client.messages.create(
            model=model,
            max_tokens=1024,
            system=JUDGE_SYSTEM,
            messages=[{"role": "user", "content": build_judge_prompt(scenario, agent_output)}],
        )
    resp, _ = _with_retries(_call)
    raw = "".join(block.text for block in resp.content if block.type == "text").strip()
    if raw.startswith("```"):
        raw = raw.split("```", 2)[1]
        if raw.startswith("json"):
            raw = raw[4:]
        raw = raw.rsplit("```", 1)[0]
    return json.loads(raw.strip())


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--trials", type=int, default=5, help="Trials per fixture (control only)")
    parser.add_argument("--fixtures", default="", help="Comma-separated fixture IDs (prefix match); empty = all")
    parser.add_argument("--agent-model", default=DEFAULT_AGENT_MODEL)
    parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
    parser.add_argument("--output", default="eval_results.json", help="Where to write the raw results")
    parser.add_argument("--fresh", action="store_true", help="Ignore an existing results file and start clean (default: resume/fill gaps)")
    # Default treatment-only: control cells are already on disk from screening and reused.
    parser.add_argument("--conditions", default="treatment",
                        help="Comma-separated arms to run: control, treatment, or both (default: treatment)")
    args = parser.parse_args()

    conditions = [c.strip() for c in args.conditions.split(",") if c.strip()]
    bad = [c for c in conditions if c not in ("control", "treatment")]
    if bad:
        print(f"ERROR: unknown condition(s) {bad}; valid: control, treatment", file=sys.stderr)
        return 2

    if "ANTHROPIC_API_KEY" not in os.environ:
        print("ERROR: ANTHROPIC_API_KEY not set", file=sys.stderr)
        return 1

    skill_md_text = ""
    if "treatment" in conditions:
        if not SKILL_MD.exists():
            print(f"ERROR: treatment arm needs a SKILL.md at {SKILL_MD}", file=sys.stderr)
            return 1
        skill_md_text = SKILL_MD.read_text()

    client = Anthropic()

    to_run = SCENARIOS
    if args.fixtures:
        filters = [f.strip() for f in args.fixtures.split(",")]
        to_run = [s for s in SCENARIOS if any(s["id"].startswith(f) for f in filters)]

    n_cells = len(to_run) * len(conditions) * args.trials
    print(f"LIFT eval [{', '.join(conditions)}]: {len(to_run)} fixtures x {len(conditions)} conditions x {args.trials} trials = {n_cells} agent calls")
    print(f"Agent model: {args.agent_model}, Judge model: {args.judge_model}\n")

    # Resume: reload any completed trials from a prior run so a re-run fills ONLY the
    # gaps (e.g. trials dropped to a transient overload), never redoing finished work.
    # Pass --fresh to ignore an existing results file and start clean.
    results: list[dict] = []
    completed: set[tuple[str, str, int]] = set()
    out_path = Path(args.output)
    if out_path.exists() and not args.fresh:
        try:
            results = json.loads(out_path.read_text())
            completed = {(r["fixture"], r["condition"], r["trial"]) for r in results}
            print(f"Resuming from {args.output}: {len(completed)} trials already complete; filling gaps only.\n")
        except (json.JSONDecodeError, KeyError, OSError):
            results, completed = [], set()

    for scenario in to_run:
        for condition in conditions:
            for trial in range(args.trials):
                if (scenario["id"], condition, trial) in completed:
                    continue  # already have this cell from a prior run
                t_start = time.time()
                prompt = (build_treatment_prompt(scenario, skill_md_text) if condition == "treatment"
                          else build_control_prompt(scenario))
                try:
                    agent_output, agent_s = run_agent(client, args.agent_model, prompt)  # agent_s excludes retry backoff
                    judge_result = run_judge(client, args.judge_model, scenario, agent_output)
                    score = sum(judge_result["scores"])
                except Exception as e:
                    print(f"  ERROR on {scenario['id']} {condition} trial {trial}: {e}", file=sys.stderr)
                    continue
                elapsed = time.time() - t_start  # agent + judge, for cost/wall-clock accounting
                results.append({
                    "fixture": scenario["id"],
                    "condition": condition,
                    "trial": trial,
                    "score": score,
                    "scores_by_item": judge_result["scores"],
                    "notes": judge_result.get("notes", []),
                    "agent_output": agent_output,
                    "agent_chars": len(agent_output),
                    "agent_s": round(agent_s, 1),
                    "elapsed_s": round(elapsed, 1),
                })
                # Crash-safe: persist after every trial via atomic temp+rename so an
                # overload-induced death never throws away completed work.
                tmp = Path(str(args.output) + ".tmp")
                tmp.write_text(json.dumps(results, indent=2))
                tmp.replace(args.output)
                print(f"  {scenario['id']:<40} | {condition:9s} | trial {trial} | score {score}/7 | audit {agent_s:.0f}s", flush=True)

    Path(args.output).write_text(json.dumps(results, indent=2))
    print(f"\nRaw results: {args.output}\n")
    print_summary(results, to_run)
    return 0


def print_summary(results: list[dict], to_run: list[dict]) -> None:
    ctrl: dict[str, list[int]] = {}
    treat: dict[str, list[int]] = {}
    for r in results:
        bucket = ctrl if r.get("condition") == "control" else treat
        bucket.setdefault(r["fixture"], []).append(r["score"])

    print(f"{'Fixture':<40} {'Control':>8} {'Treat':>8} {'Lift':>8} {'Nc':>4} {'Nt':>4}")
    print("-" * 80)
    c_means: list[float] = []
    t_means: list[float] = []
    lifts: list[float] = []
    paired_ids: list[str] = []
    for scenario in to_run:
        cs = ctrl.get(scenario["id"], [])
        ts = treat.get(scenario["id"], [])
        if not cs and not ts:
            continue
        c = statistics.mean(cs) if cs else float("nan")
        t = statistics.mean(ts) if ts else float("nan")
        c_str = f"{c:>8.2f}" if cs else f"{'n/a':>8}"
        t_str = f"{t:>8.2f}" if ts else f"{'n/a':>8}"
        if cs and ts:
            lift = t - c
            lifts.append(lift); c_means.append(c); t_means.append(t); paired_ids.append(scenario["id"])
            l_str = f"{lift:>+8.2f}"
            flag = "  <- treat still <5" if t < 5.0 else ("  <- no lift" if lift <= 0 else "")
        else:
            l_str = f"{'-':>8}"
            flag = ""
        print(f"{scenario['id']:<40} {c_str} {t_str} {l_str} {len(cs):>4} {len(ts):>4}{flag}")
    print("-" * 80)

    if not lifts:
        print("\nNo paired control/treatment fixtures to summarize "
              f"(control: {sum(len(v) for v in ctrl.values())} cells, "
              f"treatment: {sum(len(v) for v in treat.values())} cells).")
        return

    c_agg = statistics.mean(c_means)
    t_agg = statistics.mean(t_means)
    print(f"\nAggregate: control {c_agg:.2f}/7  ->  treatment {t_agg:.2f}/7   (lift {t_agg - c_agg:+.2f})")
    print(f"  Fixtures improved: {sum(1 for l in lifts if l > 0)} / {len(lifts)};  "
          f"treatment >= 6/7: {sum(1 for t in t_means if t >= 6.0)} / {len(t_means)};  "
          f"treatment >= 5/7: {sum(1 for t in t_means if t >= 5.0)} / {len(t_means)}")
    weakest = min(zip(t_means, paired_ids))
    print(f"  Weakest treatment fixture: {weakest[1]} at {weakest[0]:.2f}/7 "
          "(the next one to close with a SKILL.md edit)")


if __name__ == "__main__":
    sys.exit(main())
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall