Skip to content

anyshift-io/sre-skills / skills/iam-deceptive-escalation-auditor

iam-deceptive-escalation-auditor

Verified

FileKindSize
FAILURE_MODES.mdmarkdown3.6 KB
SKILL.mdskill md19.6 KB
fixtures/01-orphaned-passrole-deny/meta.jsonconfig349 B
fixtures/01-orphaned-passrole-deny/policy.jsonconfig573 B
fixtures/02-action-star-blanket-deny/meta.jsonconfig408 B
fixtures/02-action-star-blanket-deny/policy.jsonconfig604 B
fixtures/03-assumerole-broken-trust/meta.jsonconfig492 B
fixtures/03-assumerole-broken-trust/policy.jsonconfig492 B
fixtures/03-assumerole-broken-trust/trust-policy.jsonconfig473 B
fixtures/05-iam-mutation-boundary-capped/meta.jsonconfig696 B
fixtures/05-iam-mutation-boundary-capped/policy-1.jsonconfig550 B
fixtures/05-iam-mutation-boundary-capped/policy-2.jsonconfig945 B
fixtures/06-cross-account-assume-condition-gated/meta.jsonconfig700 B
fixtures/06-cross-account-assume-condition-gated/policy-1.jsonconfig690 B
fixtures/06-cross-account-assume-condition-gated/policy-2.jsonconfig512 B
fixtures/06-cross-account-assume-condition-gated/trust-policy.jsonconfig400 B
fixtures/07-passrole-sandboxed-role-orphaned/meta.jsonconfig997 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-1.jsonconfig508 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-2.jsonconfig571 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-3.jsonconfig640 B
fixtures/08-ml-platform-passrole-launch-needle/meta.jsonconfig1.0 KB
fixtures/08-ml-platform-passrole-launch-needle/policy-1.jsonconfig713 B
fixtures/08-ml-platform-passrole-launch-needle/policy-2.jsonconfig670 B
fixtures/08-ml-platform-passrole-launch-needle/policy-3.jsonconfig745 B
fixtures/08-ml-platform-passrole-launch-needle/policy-4.jsonconfig647 B
fixtures/08-ml-platform-passrole-launch-needle/policy-5.jsonconfig959 B
fixtures/08-ml-platform-passrole-launch-needle/policy-6.jsonconfig568 B
tests/README.mdmarkdown3.8 KB
tests/_audit.pyscript32.5 KB
tests/_replay.pyscript720 B
tests/eval/README.mdmarkdown5.5 KB
tests/eval/eval_results.jsonconfig493.4 KB
tests/eval/judge_prompt.mdmarkdown3.0 KB
tests/eval/rubric.mdmarkdown3.3 KB
tests/eval/run_eval.pyscript16.1 KB
tests/eval/scenarios.pyscript11.2 KB
tests/replay_01_orphaned_passrole_deny.pyscript1.5 KB
tests/replay_02_action_star_blanket_deny.pyscript1.4 KB
tests/replay_03_assumerole_broken_trust.pyscript1.5 KB
tests/replay_05_iam_mutation_boundary_capped.pyscript2.1 KB
tests/replay_06_cross_account_assume_condition_gated.pyscript1.8 KB
tests/replay_07_passrole_sandboxed_role_orphaned.pyscript2.0 KB
tests/replay_08_ml_platform_passrole_launch_needle.pyscript2.3 KB

tests/eval/rubric.md

View on GitHub
# Eval rubric: `iam-deceptive-escalation-auditor` (control-only screen)

Seven binary items (1 = pass, 0 = fail). No partial credit. The judge is given the deterministic reference audit (from the copied `_audit.py`, via `scenarios.py`) as ground truth.

This is a **control-only screening** rubric: there is no SKILL.md and no treatment arm. The point is to measure whether a COLD agent fails on this deliberately deceptive corpus. A skill is worth building only if the cold agent's mean score is LOW (< 4/7).

1. **Parse.** The agent reads the statements correctly and expands wildcard Actions: a `*` or `svc:*` is recognised as the concrete sensitive permissions it grants, not skimmed as "broad". It must also read `Effect: Deny` statements (and resource scopes) as constraints, not ignore them.

2. **Findings.** The agent identifies the defect(s) the ground truth lists for this fixture by substance (the same escalation path; it need not say "E1"). For a **deceptive-clean** fixture, the agent must report *no real escalation* (the apparent danger is neutralised).

3. **No false positives.** The dominant item for this corpus. On the four deceptive-clean fixtures the agent must NOT call a defect that the ground truth lacks: a `iam:PassRole` killed by an explicit `Deny`, an `Action '*'` pinned to one bucket, an `sts:AssumeRole` whose target does not trust the principal back, or a read-only `iam:Get*/List*` wildcard. Manufacturing a "critical privilege escalation" on any of these fails this item.

4. **Cross-statement reasoning.** Two directions. On the **buried-hard needles**, where the escalation is a combination split across 5 attached policies (`iam:PassRole` in one + `lambda:CreateFunction` in another; `iam:UpdateAssumeRolePolicy` + `sts:AssumeRole`; `lambda:UpdateFunctionCode` + `iam:PassRole`), the agent must evaluate the *union* of all statements and name the combo, not clear each statement in isolation. On the **deceptive-clean** fixtures, it must recognise when a `Deny` or a resource scope neutralises an apparent combo. This is the item a cold agent most often gets wrong in BOTH directions.

5. **Criticality.** The agent ranks a real escalation-to-admin (PassRole+compute, trust-rewrite+assume, function-code hijack) as the headline on the needle fixtures, and does NOT headline a neutralised or read-only grant as critical on the clean fixtures.

6. **Boundary.** The agent names at least one thing it cannot determine from the policy alone, matching the ground-truth join (the principal's other attached policies, its permissions boundary, the org SCPs, or the privileges of the role an escalation targets). An agent that presents a single policy read as a complete access verdict fails.

7. **Recommendation.** The agent's top recommended fix matches the ground-truth fix in substance (scope the action/resource, remove the escalation grant, add a narrowing condition) on the needles, or correctly states that no fix is needed because the policy is already neutralised on the clean fixtures.

A perfect audit scores 7. The cold agent is expected to lose points heavily on items 3 (over-flagging the deceptive-clean fixtures), 4 (missing the buried cross-policy combos and missing the neutralising Deny/scope), and 5 (mis-ranking severity), which is exactly why this domain is a candidate for a dedicated skill.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall