Skip to content

anyshift-io/sre-skills / skills/sg-deceptive-reachability-auditor

sg-deceptive-reachability-auditor

Unverified

FileKindSize
FAILURE_MODES.mdmarkdown3.4 KB
SKILL.mdskill md14.5 KB
fixtures/01-orphaned-front-internal-cidr/instances.jsonconfig1.2 KB
fixtures/01-orphaned-front-internal-cidr/meta.jsonconfig378 B
fixtures/01-orphaned-front-internal-cidr/security-groups.jsonconfig6.8 KB
fixtures/02-public-alb-no-sg-ref/instances.jsonconfig1.2 KB
fixtures/02-public-alb-no-sg-ref/meta.jsonconfig477 B
fixtures/02-public-alb-no-sg-ref/security-groups.jsonconfig7.1 KB
fixtures/03-disjoint-public-vpn-islands/instances.jsonconfig1.4 KB
fixtures/03-disjoint-public-vpn-islands/meta.jsonconfig569 B
fixtures/03-disjoint-public-vpn-islands/security-groups.jsonconfig6.8 KB
fixtures/04-broken-segment-midchain/instances.jsonconfig1.2 KB
fixtures/04-broken-segment-midchain/meta.jsonconfig541 B
fixtures/04-broken-segment-midchain/security-groups.jsonconfig6.3 KB
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/instances.jsonconfig1.4 KB
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/meta.jsonconfig394 B
fixtures/05-six-hop-cdn-waf-gw-app-svc-db/security-groups.jsonconfig8.0 KB
fixtures/06-compromised-ci-runner-deep/instances.jsonconfig1.6 KB
fixtures/06-compromised-ci-runner-deep/meta.jsonconfig466 B
fixtures/06-compromised-ci-runner-deep/security-groups.jsonconfig7.5 KB
fixtures/07-five-hop-ingress-mesh-broker-db/instances.jsonconfig1.2 KB
fixtures/07-five-hop-ingress-mesh-broker-db/meta.jsonconfig394 B
fixtures/07-five-hop-ingress-mesh-broker-db/security-groups.jsonconfig7.4 KB
tests/README.mdmarkdown3.1 KB
tests/_deep.pyscript3.0 KB
tests/_reach_engine.pyscript20.9 KB
tests/_replay.pyscript881 B
tests/eval/README.mdmarkdown4.3 KB
tests/eval/eval_results.jsonconfig561.5 KB
tests/eval/judge_prompt.mdmarkdown3.2 KB
tests/eval/rubric.mdmarkdown4.7 KB
tests/eval/run_eval.pyscript19.8 KB
tests/eval/scenarios.pyscript13.2 KB
tests/replay_01_orphaned_front_internal_cidr.pyscript1.9 KB
tests/replay_02_public_alb_no_sg_ref.pyscript1.9 KB
tests/replay_03_disjoint_public_vpn_islands.pyscript2.0 KB
tests/replay_04_broken_segment_midchain.pyscript2.0 KB
tests/replay_05_six_hop_cdn_waf_gw_app_svc_db.pyscript1.8 KB
tests/replay_06_compromised_ci_runner_deep.pyscript2.0 KB
tests/replay_07_five_hop_ingress_mesh_broker_db.pyscript1.8 KB

tests/eval/judge_prompt.md

View on GitHub
# Judge prompt (reference)

This file documents the judge contract. The runner (`run_eval.py`) builds the live judge
prompt programmatically from this template + the deterministic ground truth from
`scenarios.expected_deep()`. Kept here so the rubric anchoring is reviewable without
reading Python.

## System

> You are an expert AWS / cloud-security evaluator grading a fleet security review against
> a 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial
> credit. You will be given a known-good answer from a deterministic reference audit, the
> agent's review output, and the 7 rubric items. Return JSON only (no prose):
>
> ```json
> {"scores": [int, int, int, int, int, int, int], "notes": ["one sentence", ...]}
> ```

## User (per trial)

Provides:

- **FIXTURE** id and **CONTEXT** (the generic fleet description the agent saw).
- **KNOWN-GOOD ANSWER** from the reused deterministic engine:
  - ground-truth finding codes (P1 path / B1 blast radius / H1 hub), top severity, clean flag,
    SG count, SG-to-SG edge count;
  - the shortest path entry -> crown jewel (or "(no path)"), with its hop count;
  - the blast radius (reachable tiers from the entry);
  - the expected headline (what the buried defect actually is, or that it is clean);
  - the correct top fix; and
  - the boundary the audit cannot cross.
- **AGENT REVIEW OUTPUT** (verbatim).
- The **7 RUBRIC ITEMS** (see `rubric.md`), with the two load-bearing items called out:
  item 2 (surfaces the buried LONG 4-6 hop needle as a primary finding, hops named end to
  end, not only the obvious surface items) and item 3 (does not over-flag the benign /
  neutralised bait, esp. the orphaned deep chain in `01`, the intended public ALB in `02`,
  the disconnected data island in `03`, and the broken chain in `04`).

## Anchoring rules for the judge

- The agent prompt was GENERIC ("review this for problems"); it did NOT name lateral
  movement, reachability, multi-hop depth, or the crown-jewel path. Reward the agent only
  for what it surfaces on its own. Do not penalize phrasing; grade substance.
- The needle is the composed LONG (4-6 hop) SG-to-SG path. Reading each rule as locally
  fine and stopping, or stopping at a single hop, is an item-2 and item-4 fail even if every
  individual observation is true. Naming only some hops but not assembling the full chain to
  the crown jewel does not satisfy item 2.
- On the deceptive-clean fixtures (`01`, `02`, `03`, `04`), the correct answer is "no
  reachable path to the crown jewel." Inventing a path from an orphaned deep chain (`01`),
  headlining the intended public-ALB `0.0.0.0/0` as the lateral exposure (`02`), splicing two
  disconnected islands into one route (`03`), or treating the broken chain as reachable across
  the cut (`04`), fails items 2/3/5. The agent must reason about reachability — note that the
  front tier accepts an internal CIDR rather than the ALB SG, or that nothing references the
  ALB SG, or that the islands have no joining edge — not just list the rules.
- Match the top fix and the boundary by substance, not wording. On a clean fixture, the
  correct top fix is "no change beyond confirming the boundary," NOT a fix for a path that
  does not exist.

Return JSON only.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall