Skip to content

anyshift-io/sre-skills / skills/iam-deceptive-escalation-auditor

iam-deceptive-escalation-auditor

Verified

FileKindSize
FAILURE_MODES.mdmarkdown3.6 KB
SKILL.mdskill md19.6 KB
fixtures/01-orphaned-passrole-deny/meta.jsonconfig349 B
fixtures/01-orphaned-passrole-deny/policy.jsonconfig573 B
fixtures/02-action-star-blanket-deny/meta.jsonconfig408 B
fixtures/02-action-star-blanket-deny/policy.jsonconfig604 B
fixtures/03-assumerole-broken-trust/meta.jsonconfig492 B
fixtures/03-assumerole-broken-trust/policy.jsonconfig492 B
fixtures/03-assumerole-broken-trust/trust-policy.jsonconfig473 B
fixtures/05-iam-mutation-boundary-capped/meta.jsonconfig696 B
fixtures/05-iam-mutation-boundary-capped/policy-1.jsonconfig550 B
fixtures/05-iam-mutation-boundary-capped/policy-2.jsonconfig945 B
fixtures/06-cross-account-assume-condition-gated/meta.jsonconfig700 B
fixtures/06-cross-account-assume-condition-gated/policy-1.jsonconfig690 B
fixtures/06-cross-account-assume-condition-gated/policy-2.jsonconfig512 B
fixtures/06-cross-account-assume-condition-gated/trust-policy.jsonconfig400 B
fixtures/07-passrole-sandboxed-role-orphaned/meta.jsonconfig997 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-1.jsonconfig508 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-2.jsonconfig571 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-3.jsonconfig640 B
fixtures/08-ml-platform-passrole-launch-needle/meta.jsonconfig1.0 KB
fixtures/08-ml-platform-passrole-launch-needle/policy-1.jsonconfig713 B
fixtures/08-ml-platform-passrole-launch-needle/policy-2.jsonconfig670 B
fixtures/08-ml-platform-passrole-launch-needle/policy-3.jsonconfig745 B
fixtures/08-ml-platform-passrole-launch-needle/policy-4.jsonconfig647 B
fixtures/08-ml-platform-passrole-launch-needle/policy-5.jsonconfig959 B
fixtures/08-ml-platform-passrole-launch-needle/policy-6.jsonconfig568 B
tests/README.mdmarkdown3.8 KB
tests/_audit.pyscript32.5 KB
tests/_replay.pyscript720 B
tests/eval/README.mdmarkdown5.5 KB
tests/eval/eval_results.jsonconfig493.4 KB
tests/eval/judge_prompt.mdmarkdown3.0 KB
tests/eval/rubric.mdmarkdown3.3 KB
tests/eval/run_eval.pyscript16.1 KB
tests/eval/scenarios.pyscript11.2 KB
tests/replay_01_orphaned_passrole_deny.pyscript1.5 KB
tests/replay_02_action_star_blanket_deny.pyscript1.4 KB
tests/replay_03_assumerole_broken_trust.pyscript1.5 KB
tests/replay_05_iam_mutation_boundary_capped.pyscript2.1 KB
tests/replay_06_cross_account_assume_condition_gated.pyscript1.8 KB
tests/replay_07_passrole_sandboxed_role_orphaned.pyscript2.0 KB
tests/replay_08_ml_platform_passrole_launch_needle.pyscript2.3 KB

tests/eval/README.md

View on GitHub
# Control-only screening eval

Is a dedicated `iam-deceptive-escalation-auditor` skill worth building? Only if a COLD agent (no skill, generic "review this policy for problems" prompt) already fails on this domain. This harness measures exactly that. There is **no SKILL.md and no treatment arm**: it runs the control condition only and reports whether the cold agent scores LOW.

The replay tests under `tests/replay_*.py` prove the reference engine produces the intended verdict on every fixture (six deceptive-clean, one buried-hard needle). This eval then runs a live agent against the same fixtures and scores it against a 7-item rubric ([`rubric.md`](./rubric.md)), anchored to that engine as ground truth.

**Screening rule:** build the skill only if the aggregate control mean is **< 4 / 7**. A high control score means the base model already handles deceptive IAM escalation and a skill adds little.

## Quickstart

```bash
# Install the only non-stdlib dependency
pip install anthropic

# Set your API key
export ANTHROPIC_API_KEY=sk-ant-...

# Smoke test: 1 trial, 3 fixtures (two clean, one needle)
python tests/eval/run_eval.py --trials 1 --fixtures 02,05,08

# Full screen: 5 trials, all 7 fixtures (~70 LLM calls)
python tests/eval/run_eval.py --trials 5
```

The script writes raw per-trial results to `eval_results.json` and prints a per-fixture summary table with the aggregate control mean and a screening verdict. `python tests/eval/scenarios.py` prints the deterministic ground-truth findings for every fixture with **no API key required**.

## Resume / crash-safety

The eval is **resumable, and each trial is independent**. Every completed `(fixture, condition, trial)` cell is written to `eval_results.json` immediately via an atomic temp-file rename, and on startup the runner reloads what is already on disk and runs **only the missing cells**. So:

- An interrupt (`Ctrl-C`), a crash, or an API overload mid-run never throws away completed work.
- To finish a partial run, **re-run the exact same command** — it fills the gaps and stops.
- `run_eval.py` also wraps every API call in exponential backoff (`_with_retries`) over 429 / 5xx / "overloaded" errors.
- Pass `--fresh` to ignore an existing results file and start clean.

## The fixtures (why they are all in the cold agent's weak region)

Every fixture is engineered so a single-statement read gives the wrong answer. None is an obvious admin-star the base model trivially flags.

| Fixture | Engine verdict | The trap |
|---|---|---|
| 01 orphaned-passrole-deny | clean | PassRole + RunInstances both present, but an explicit `Deny` on `iam:PassRole` kills the combo. |
| 02 action-star-blanket-deny | clean | `Action '*'` pinned to one sandbox bucket (never Resource `*`), plus a `Deny` on every escalation service. |
| 03 assumerole-broken-trust | clean | `sts:AssumeRole` on an admin-sounding role whose trust does not point back; no `UpdateAssumeRolePolicy` to fix it. |
| 05 iam-mutation-boundary-capped | clean | A full mutation kit (PutRolePolicy/Attach/CreatePolicyVersion/UpdateAssumeRolePolicy/PassRole/CreateAccessKey) scoped to one break-glass ARN, fully capped by a permission-boundary-style `Deny` across Resource `*`. |
| 06 cross-account-assume-condition-gated | clean | A cross-account `sts:AssumeRole` that looks like a pivot but is sealed by an `aws:PrincipalOrgID` + `sts:ExternalId` Condition the principal cannot satisfy; the target trust's wildcard Principal is narrowed by the same condition (no X1). |
| 07 passrole-sandboxed-role-orphaned | clean | `iam:PassRole` + compute verbs (Start/Invoke) read like the launch combo, but the verbs bind no role and the one passable role is read-only: an orphaned escalation with no privilege gain. |
| 08 ml-platform-passrole-launch-needle | E1 (critical) | `iam:PassRole` (`*`) and `sagemaker:CreateTrainingJob` split four policies apart across six attached policies of benign ML-platform bait. |

The cold agent is expected to over-flag the six clean fixtures as critical privilege escalation (item 3), the proven capability gap, and to miss the one buried combo by reading each statement in isolation (item 4).

## What the eval does

For each fixture, the script runs N trials in the control condition: the agent is given the raw IAM policy JSON (permissions policy, plus trust where present) and a generic "audit this policy for misconfigurations" prompt. It uses whatever it brings from training. Each output is graded by an LLM judge against the 7-item rubric, anchored to the deterministic reference audit (`_audit.py`, via `scenarios.py`) as ground truth, so grading is against a known-good answer rather than the judge's own opinion.

## Files

| File | Purpose |
|---|---|
| `run_eval.py` | The control-only runner. Calls the API once per trial, calls the judge, aggregates. Resumable. Needs `ANTHROPIC_API_KEY`. |
| `scenarios.py` | Per-fixture contexts and ground-truth findings (computed by importing the copied `_audit.py`, so they never drift). Runs offline. |
| `rubric.md` | The 7-item rubric, one sentence per item. |
| `judge_prompt.md` | The judge prompt template, for swapping in a different judge. |

## What the eval does NOT measure

- **Lift.** There is no treatment arm; this screen measures only whether the cold baseline is weak enough to justify building the skill.
- **Narrative quality / speed / cost.** Not part of the score.
- **Real-world generalization.** The seven fixtures are constructed, not pulled from production. A low control score here is a *signal* that the domain is hard for a cold agent, not proof of operational value.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall