Skip to content

anyshift-io/sre-skills / skills/sg-deceptive-reachability-auditor

sg-deceptive-reachability-auditor

Unverified

FileKindSize
FAILURE_MODES.mdmarkdown3.4 KB
SKILL.mdskill md14.5 KB
fixtures/01-orphaned-front-internal-cidr/instances.jsonconfig1.2 KB
fixtures/01-orphaned-front-internal-cidr/meta.jsonconfig378 B
fixtures/01-orphaned-front-internal-cidr/security-groups.jsonconfig6.8 KB
fixtures/02-public-alb-no-sg-ref/instances.jsonconfig1.2 KB
fixtures/02-public-alb-no-sg-ref/meta.jsonconfig477 B
fixtures/02-public-alb-no-sg-ref/security-groups.jsonconfig7.1 KB
fixtures/03-disjoint-public-vpn-islands/instances.jsonconfig1.4 KB
fixtures/03-disjoint-public-vpn-islands/meta.jsonconfig569 B
fixtures/03-disjoint-public-vpn-islands/security-groups.jsonconfig6.8 KB
fixtures/04-broken-segment-midchain/instances.jsonconfig1.2 KB
fixtures/04-broken-segment-midchain/meta.jsonconfig541 B
fixtures/04-broken-segment-midchain/security-groups.jsonconfig6.3 KB
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/instances.jsonconfig1.4 KB
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/meta.jsonconfig394 B
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/security-groups.jsonconfig8.0 KB
fixtures/06-compromised-ci-runner-deep/instances.jsonconfig1.6 KB
fixtures/06-compromised-ci-runner-deep/meta.jsonconfig466 B
fixtures/06-compromised-ci-runner-deep/security-groups.jsonconfig7.5 KB
fixtures/07-five-hop-ingress-mesh-broker-db/instances.jsonconfig1.2 KB
fixtures/07-five-hop-ingress-mesh-broker-db/meta.jsonconfig394 B
fixtures/07-five-hop-ingress-mesh-broker-db/security-groups.jsonconfig7.4 KB
tests/README.mdmarkdown3.1 KB
tests/_deep.pyscript3.0 KB
tests/_reach_engine.pyscript20.9 KB
tests/_replay.pyscript881 B
tests/eval/README.mdmarkdown4.3 KB
tests/eval/eval_results.jsonconfig561.5 KB
tests/eval/judge_prompt.mdmarkdown3.2 KB
tests/eval/rubric.mdmarkdown4.7 KB
tests/eval/run_eval.pyscript19.8 KB
tests/eval/scenarios.pyscript13.2 KB
tests/replay_01_orphaned_front_internal_cidr.pyscript1.9 KB
tests/replay_02_public_alb_no_sg_ref.pyscript1.9 KB
tests/replay_03_disjoint_public_vpn_islands.pyscript2.0 KB
tests/replay_04_broken_segment_midchain.pyscript2.0 KB
tests/replay_05_six_hop_cdn_waf_gw_app_svc_db.pyscript1.8 KB
tests/replay_06_compromised_ci_runner_deep.pyscript2.0 KB
tests/replay_07_five_hop_ingress_mesh_broker_db.pyscript1.8 KB

tests/eval/rubric.md

View on GitHub
# Rubric: sg-deceptive-reachability-auditor (control-only screening)

Seven binary items. Each is 1 (pass) or 0 (fail); no partial credit. The judge grades the
cold agent's audit of a high-volume fleet (10-13 security groups) against the deterministic
ground truth from the reused reachability engine (`_deep.py` -> `_reach_engine.py`).

The agent was given a GENERIC prompt: the raw `describe-security-groups` +
`describe-instances` JSON for the whole fleet, the named entry point and crown-jewel tier,
and "review this for security/risk problems, rank by severity, give the top fix, and say
what you cannot determine." The prompt did NOT name lateral movement, reachability, the
SG-to-SG chain, multi-hop depth, or the crown-jewel path. The screening question is whether
the cold agent composes the non-obvious LONG (4-6 hop) needle in the haystack unprompted, and
whether it stays quiet on the DECEPTIVE/segmented-clean fleets instead of fabricating a path.

Every fixture is from the model's empirically-located WEAK region: there are NO short,
obvious 2-3 hop direct paths and NO single obvious 0.0.0.0/0 -> db rule (those the base model
already aces). Three fixtures are 5-6 hop needles; four are deceptive-clean fleets that bait
with a loud public CIDR or a visible-but-orphaned / broken deep SG-ref chain that does NOT
reach the crown jewel.

1. **Parse.** Recognises `UserIdGroupPairs` as SG-to-SG edges (SG B accepting SG A means a
   host in A can reach B), reads `0.0.0.0/0` / `::/0` as the internet entry (or the named
   compromised host as the entry, fixture `06`), reads the instance->SG membership map, and
   does not skip the SG-reference arrays in the volume.

2. **Surfaces the buried non-obvious issue (LOAD-BEARING).** On a needle fixture, the agent
   surfaces the buried LONG (4-6 hop) SG-to-SG reachable path to the crown jewel as A (the)
   PRIMARY finding, with the hops named end to end, rather than only listing the obvious
   surface items (a public `0.0.0.0/0` rule, a broad port range, a single hop) and stopping.
   On a deceptive-clean fixture, it correctly reports no real reachable path to the crown
   jewel. This is the item the cold agent most often misses: in a 10-13 SG haystack it tends
   to flag the loud CIDR exposure, clear each rule locally, and never compose the quiet long
   SG-ref chain.

3. **Does not over-flag the benign / neutralised bait (LOAD-BEARING).** Does not assert a
   path or exposure the ground truth lacks. Specifically: does NOT report the ORPHANED deep
   chain in `01` as a reachable path; does NOT call the public ALB's intended `0.0.0.0/0` in
   `02` the lateral path / the headline; does NOT splice the disconnected data island in `03`
   into a manufactured internet->db route; does NOT treat the BROKEN chain in `04` as
   reachable across the cut; does NOT manufacture any internet->db route on a clean fleet; and
   does NOT drown the real finding (or the clean verdict) in a wall of low-value nitpicks
   about correctly-scoped tiers (bastion, monitoring, ci, ssm).

4. **Composition / multi-hop (LONG).** Traces the SG-to-SG edges into a transitive LONG
   (4-6 hop) path rather than clearing each rule / SG in isolation. The agent must assemble
   entry -> ... -> crown jewel across many separate SGs end to end, not list each SG as
   locally fine and stop, and not stop at a single hop. On the deceptive-clean fleets, the
   same composition discipline is what reveals the chain is orphaned / broken / cross-island,
   so the agent must reason about reachability, not just rule locality.

5. **Criticality.** Ranks the long path that reaches the crown jewel as the headline
   (critical), the blast radius as high. Does NOT headline the directly-internet-facing
   front-door tier, nor (in `02`/`04`) the loud public-ALB / edge-proxy rule, over the quiet
   real long path. On a deceptive-clean fixture, does not invent a critical.

6. **Boundary.** Names at least one thing it cannot determine from the SG graph + membership
   alone, matching the ground-truth join: reachability is not exploitability -- live SG
   membership / running hosts, route tables, NACLs, or app-layer auth.

7. **Recommendation.** The top fix matches the ground truth in substance: break the
   offending edge/hop on the long chain (or interpose a broker/bastion), or (on a
   deceptive-clean fixture) no change beyond confirming the boundary — explicitly NOT
   "fix" a path that does not exist.

## Verdict (computed from CONTROL means only)

- Aggregate control mean **< 4.0/7**, or a **majority** of fixtures below 4.0 -> **BUILD**
  (cold agent is weak here; the skill is worth writing).
- Aggregate **< 5.5/7** -> **MAYBE** (mixed; inspect per-fixture, especially items 2 and 4).
- Otherwise -> **SKIP** (cold agent already strong; the skill adds little).
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall