Skip to content

anyshift-io/sre-skills / skills/sg-deceptive-reachability-auditor

sg-deceptive-reachability-auditor

Unverified

FileKindSize
FAILURE_MODES.mdmarkdown3.4 KB
SKILL.mdskill md14.5 KB
fixtures/01-orphaned-front-internal-cidr/instances.jsonconfig1.2 KB
fixtures/01-orphaned-front-internal-cidr/meta.jsonconfig378 B
fixtures/01-orphaned-front-internal-cidr/security-groups.jsonconfig6.8 KB
fixtures/02-public-alb-no-sg-ref/instances.jsonconfig1.2 KB
fixtures/02-public-alb-no-sg-ref/meta.jsonconfig477 B
fixtures/02-public-alb-no-sg-ref/security-groups.jsonconfig7.1 KB
fixtures/03-disjoint-public-vpn-islands/instances.jsonconfig1.4 KB
fixtures/03-disjoint-public-vpn-islands/meta.jsonconfig569 B
fixtures/03-disjoint-public-vpn-islands/security-groups.jsonconfig6.8 KB
fixtures/04-broken-segment-midchain/instances.jsonconfig1.2 KB
fixtures/04-broken-segment-midchain/meta.jsonconfig541 B
fixtures/04-broken-segment-midchain/security-groups.jsonconfig6.3 KB
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/instances.jsonconfig1.4 KB
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/meta.jsonconfig394 B
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/security-groups.jsonconfig8.0 KB
fixtures/06-compromised-ci-runner-deep/instances.jsonconfig1.6 KB
fixtures/06-compromised-ci-runner-deep/meta.jsonconfig466 B
fixtures/06-compromised-ci-runner-deep/security-groups.jsonconfig7.5 KB
fixtures/07-five-hop-ingress-mesh-broker-db/instances.jsonconfig1.2 KB
fixtures/07-five-hop-ingress-mesh-broker-db/meta.jsonconfig394 B
fixtures/07-five-hop-ingress-mesh-broker-db/security-groups.jsonconfig7.4 KB
tests/README.mdmarkdown3.1 KB
tests/_deep.pyscript3.0 KB
tests/_reach_engine.pyscript20.9 KB
tests/_replay.pyscript881 B
tests/eval/README.mdmarkdown4.3 KB
tests/eval/eval_results.jsonconfig561.5 KB
tests/eval/judge_prompt.mdmarkdown3.2 KB
tests/eval/rubric.mdmarkdown4.7 KB
tests/eval/run_eval.pyscript19.8 KB
tests/eval/scenarios.pyscript13.2 KB
tests/replay_01_orphaned_front_internal_cidr.pyscript1.9 KB
tests/replay_02_public_alb_no_sg_ref.pyscript1.9 KB
tests/replay_03_disjoint_public_vpn_islands.pyscript2.0 KB
tests/replay_04_broken_segment_midchain.pyscript2.0 KB
tests/replay_05_six_hop_cdn_waf_gw_app_svc_db.pyscript1.8 KB
tests/replay_06_compromised_ci_runner_deep.pyscript2.0 KB
tests/replay_07_five_hop_ingress_mesh_broker_db.pyscript1.8 KB

tests/README.md

View on GitHub
# Tests: sg-deceptive-reachability-auditor

Deterministic ground-truth + replay tests for the deceptive-reachability screening fixtures.
No API key, no network, stdlib only.

## Engine (reused, not re-derived)

- `_reach_engine.py` — a **verbatim byte-for-byte copy** of the validated engine from the
  sibling skill `lateral-movement-reachability-auditor/tests/_reach.py` (carried in via the
  `sg-deep-lateral-auditor` screening harness). The graph build, BFS transitive closure,
  shortest-path, and articulation-hub logic are unchanged, so the ground truth here is
  provably the same computation.
- `_deep.py` — thin wrapper that re-exports the engine and exposes
  `run_deep(fixture_dir) -> Reachability`. It does NOT change engine logic. The
  `Reachability` result exposes `.findings` / `.codes()` / `.top_severity` / `.clean` /
  `.boundary` / `.shortest_path` / `.reachable`. For this fleet/estate harness, the
  "aggregation across sub-items" is the single fleet-wide transitive closure the engine
  already computes over the whole SG set (every SG a node, every UserIdGroupPair an edge).

## Scope: the model's WEAK region only

Every fixture is from the empirically-located weak region: LONG (4-6 hop) lateral chains
buried in a 10-13 SG haystack, plus DECEPTIVE / segmented-clean fleets where a loud public
exposure or a visible-but-orphaned deep SG-ref chain must NOT be reported as a reachable
path. There are **no short, obvious 2-3 hop direct paths** and **no fixture where the issue
is a single obvious public-to-db rule** — the base model trivially nails those (7.0), and
including them would lift the screening aggregate above threshold and defeat the purpose.

## Replay tests

One per fixture; each asserts the ground-truth verdict (long needle path / deceptive-clean):

```bash
for f in tests/replay_*.py; do python3 "$f"; done
```

| Fixture | Verdict | Needle / why clean |
|---|---|---|
| `01-orphaned-front-internal-cidr` | CLEAN | deep web->app->cache->db chain orphaned (web accepts the internal mesh CIDR, not the ALB SG) |
| `02-public-alb-no-sg-ref` | CLEAN | loud public ALB intended; nothing references the ALB SG (forwards by IP) |
| `03-disjoint-public-vpn-islands` | CLEAN | two disconnected islands; the data island's deep chain is VPN-only |
| `04-broken-segment-midchain` | CLEAN | visible edge->web->app->cache->db chain cut one hop in (web accepts the mesh CIDR, not the edge SG) |
| `05-six-hop-cdn-waf-gw-app-svc-db` | P1 + B1 + H1 | internet -> cdn -> waf -> gw -> app -> svc -> db (6 hops) |
| `06-compromised-ci-runner-deep` | P1 + B1 | ci -> build -> artifact -> deploy -> app -> db (5 hops, compromised-host entry) |
| `07-five-hop-ingress-mesh-broker-db` | P1 + B1 + H1 | internet -> ingress -> mesh -> app -> broker -> db (5 hops) |

Mix: 4 deceptive/segmented-clean, 3 buried-deep needles. Each fixture is a high-volume fleet
(10-13 SGs).

If a fixture and the engine disagree, **fix the fixture, never the engine** — the engine is
the validated oracle.

## Eval

`tests/eval/` holds the control-vs-treatment lift eval that measures the `SKILL.md`. See `tests/eval/README.md`.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall