Skip to content

anyshift-io/sre-skills / skills/iam-deceptive-escalation-auditor

iam-deceptive-escalation-auditor

Verified

FileKindSize
FAILURE_MODES.mdmarkdown3.6 KB
SKILL.mdskill md19.6 KB
fixtures/01-orphaned-passrole-deny/meta.jsonconfig349 B
fixtures/01-orphaned-passrole-deny/policy.jsonconfig573 B
fixtures/02-action-star-blanket-deny/meta.jsonconfig408 B
fixtures/02-action-star-blanket-deny/policy.jsonconfig604 B
fixtures/03-assumerole-broken-trust/meta.jsonconfig492 B
fixtures/03-assumerole-broken-trust/policy.jsonconfig492 B
fixtures/03-assumerole-broken-trust/trust-policy.jsonconfig473 B
fixtures/05-iam-mutation-boundary-capped/meta.jsonconfig696 B
fixtures/05-iam-mutation-boundary-capped/policy-1.jsonconfig550 B
fixtures/05-iam-mutation-boundary-capped/policy-2.jsonconfig945 B
fixtures/06-cross-account-assume-condition-gated/meta.jsonconfig700 B
fixtures/06-cross-account-assume-condition-gated/policy-1.jsonconfig690 B
fixtures/06-cross-account-assume-condition-gated/policy-2.jsonconfig512 B
fixtures/06-cross-account-assume-condition-gated/trust-policy.jsonconfig400 B
fixtures/07-passrole-sandboxed-role-orphaned/meta.jsonconfig997 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-1.jsonconfig508 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-2.jsonconfig571 B
fixtures/07-passrole-sandboxed-role-orphaned/policy-3.jsonconfig640 B
fixtures/08-ml-platform-passrole-launch-needle/meta.jsonconfig1.0 KB
fixtures/08-ml-platform-passrole-launch-needle/policy-1.jsonconfig713 B
fixtures/08-ml-platform-passrole-launch-needle/policy-2.jsonconfig670 B
fixtures/08-ml-platform-passrole-launch-needle/policy-3.jsonconfig745 B
fixtures/08-ml-platform-passrole-launch-needle/policy-4.jsonconfig647 B
fixtures/08-ml-platform-passrole-launch-needle/policy-5.jsonconfig959 B
fixtures/08-ml-platform-passrole-launch-needle/policy-6.jsonconfig568 B
tests/README.mdmarkdown3.8 KB
tests/_audit.pyscript32.5 KB
tests/_replay.pyscript720 B
tests/eval/README.mdmarkdown5.5 KB
tests/eval/eval_results.jsonconfig493.4 KB
tests/eval/judge_prompt.mdmarkdown3.0 KB
tests/eval/rubric.mdmarkdown3.3 KB
tests/eval/run_eval.pyscript16.1 KB
tests/eval/scenarios.pyscript11.2 KB
tests/replay_01_orphaned_passrole_deny.pyscript1.5 KB
tests/replay_02_action_star_blanket_deny.pyscript1.4 KB
tests/replay_03_assumerole_broken_trust.pyscript1.5 KB
tests/replay_05_iam_mutation_boundary_capped.pyscript2.1 KB
tests/replay_06_cross_account_assume_condition_gated.pyscript1.8 KB
tests/replay_07_passrole_sandboxed_role_orphaned.pyscript2.0 KB
tests/replay_08_ml_platform_passrole_launch_needle.pyscript2.3 KB

tests/eval/judge_prompt.md

View on GitHub
# Judge prompt template (control-only screen)

The judge is an LLM given the deterministic reference audit (the copied `_audit.py`, via `scenarios.py`) as ground truth, then asked to score one agent output against the 7-item [`rubric.md`](./rubric.md). It returns JSON only.

The live template is built in `run_eval.py` (`JUDGE_SYSTEM` + `build_judge_prompt`). It is reproduced here so a contributor can swap in a different judge model, or grade by hand, without reading the runner.

## System prompt

```
You are an expert AWS / cloud-security evaluator grading an IAM policy audit against a
7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial
credit.

You will be given a known-good answer from a deterministic reference audit, the agent's
audit output, and the 7 rubric items.

Return JSON only (no prose), with this exact schema:

{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence>", ...]
}
```

## User prompt

```
FIXTURE: <id>
PRINCIPAL: <principal>

KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): <codes>  (severity: <top_severity>; clean: <clean>)
- What the defect is: <expected_headline>
- Correct top fix: <expected_top_fix>
- Boundary the audit cannot cross: <expected_boundary_join>

AGENT AUDIT OUTPUT:
<agent_output>

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: ... (wildcard Actions expanded; Deny and resource scope read as constraints)
2. Findings: ... (the ground-truth defect by substance, or NO real escalation for a deceptive-clean fixture)
3. No false positives: ... (no invented defect; a Deny-neutralised / scoped / broken-trust / read-only grant is NOT a critical escalation)
4. Cross-statement reasoning: ... (unions statements to catch a buried combo; recognises a Deny/scope that neutralises an apparent combo)
5. Criticality: ... (real escalation is the headline; a neutralised/read-only grant is not critical)
6. Boundary: ... (names a join it cannot make from the policy alone)
7. Recommendation: ... (top fix matches the ground-truth fix, or "no fix needed" for a neutralised policy)

Return JSON only.
```

The placeholders in angle brackets are filled per fixture from `scenarios.py`. The full rubric text is in [`rubric.md`](./rubric.md); the runner inlines a one-line version of each item.

## Why anchor the judge to the reference audit

Without a ground-truth anchor, an LLM judge grades against its own opinion of what an IAM audit should say, which is exactly the thing under test. Feeding it the deterministic `_audit.py` findings (the same ones the replay tests assert) makes the judge score *agreement with a known-good answer* rather than *its own re-derivation*. This matters most on the deceptive-clean fixtures: the anchor tells the judge the correct answer is "no real escalation", so an agent that over-flags is scored against the truth, not against the judge's own (possibly equally over-eager) instinct. The judge can still be wrong; spot-check graded outputs to calibrate trust.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall