Skip to content

anyshift-io/sre-skills / skills/sqs-queue-auditor

sqs-queue-auditor

Verified

FileKindSize
FAILURE_MODES.mdmarkdown6.6 KB
README.mdmarkdown4.9 KB
SKILL.mdskill md13.3 KB
examples/01-no-dlq.mdmarkdown2.9 KB
examples/02-dlq-retention-shorter-than-source.mdmarkdown3.3 KB
examples/03-maxreceivecount-too-low.mdmarkdown2.8 KB
examples/04-poison-ages-out-before-dlq.mdmarkdown3.9 KB
examples/05-default-visibility-short-retention.mdmarkdown3.2 KB
examples/06-public-queue-policy.mdmarkdown3.6 KB
examples/07-fifo-dedup-off.mdmarkdown2.8 KB
examples/08-clean-standard.mdmarkdown2.6 KB
fixtures/01-no-dlq/queue.jsonconfig564 B
fixtures/02-dlq-retention-shorter-than-source/dlq.jsonconfig649 B
fixtures/02-dlq-retention-shorter-than-source/queue.jsonconfig632 B
fixtures/03-maxreceivecount-too-low/dlq.jsonconfig659 B
fixtures/03-maxreceivecount-too-low/queue.jsonconfig637 B
fixtures/04-poison-ages-out-before-dlq/dlq.jsonconfig663 B
fixtures/04-poison-ages-out-before-dlq/queue.jsonconfig648 B
fixtures/05-default-visibility-short-retention/dlq.jsonconfig650 B
fixtures/05-default-visibility-short-retention/queue.jsonconfig631 B
fixtures/06-public-queue-policy/dlq.jsonconfig663 B
fixtures/06-public-queue-policy/queue.jsonconfig908 B
fixtures/07-fifo-dedup-off/dlq.jsonconfig748 B
fixtures/07-fifo-dedup-off/queue.jsonconfig804 B
fixtures/08-clean-standard/dlq.jsonconfig672 B
fixtures/08-clean-standard/queue.jsonconfig1.0 KB
tests/README.mdmarkdown4.2 KB
tests/_audit.pyscript21.8 KB
tests/_replay.pyscript720 B
tests/eval/README.mdmarkdown8.4 KB
tests/eval/eval_results.jsonconfig220.4 KB
tests/eval/judge_prompt.mdmarkdown1.4 KB
tests/eval/queues.pyscript7.6 KB
tests/eval/rubric.mdmarkdown2.6 KB
tests/eval/run_eval.pyscript11.5 KB
tests/replay_01_no_dlq.pyscript1.5 KB
tests/replay_02_dlq_retention.pyscript1.5 KB
tests/replay_03_maxreceivecount.pyscript1.3 KB
tests/replay_04_poison_ages_out.pyscript1.6 KB
tests/replay_05_default_visibility.pyscript1.8 KB
tests/replay_06_public_policy.pyscript1.4 KB
tests/replay_07_fifo_dedup_off.pyscript1.4 KB
tests/replay_08_clean_standard.pyscript1.6 KB

tests/README.md

View on GitHub
# Replay tests for `sqs-queue-auditor`

Stdlib-only Python tests that exercise the audit in [`../SKILL.md`](../SKILL.md) against committed fixtures. No external credentials required.

## Running the tests

From the skill directory (`skills/sqs-queue-auditor/`):

```bash
for t in tests/replay_*.py; do python "$t" || exit 1; done
```

Each test prints `PASS` or `FAIL` and exits with the appropriate code. The current suite has 8 tests covering all nine rules (R1-R9), the severity model, and a clean control that asserts no false positives, totalling 48 assertions. Wire them into CI as plain `python` invocations.

## What the tests assert

Each replay test loads the fixtures for one worked example, runs the reference audit (`_audit.py`) against them, and asserts:

- The queue is parsed correctly (ARN, FIFO flag, DLQ presence).
- The expected rule(s) fire, and only those (each example isolates a rule, except where two genuinely co-occur).
- The severity is correct (the critical rules R3 / R4 are asserted critical; the soft flags R5 / R8 / R9 are asserted low).
- The boundary is reported, and names the specific join the example depends on (consumer time, IAM union, producer contract).

A test fails when the audit regresses on any of these. Treat a failed replay test as a regression in `SKILL.md` or in the reference implementation, not a test bug.

## Fixture schema

Each example has its own fixture directory under `../fixtures/<example-slug>/`. Files mirror the real AWS `GetQueueAttributes` response: a single JSON object with an `Attributes` map whose values are **all strings**, and whose compound attributes (`RedrivePolicy`, `Policy`, `RedriveAllowPolicy`) are JSON documents **encoded as strings**.

| File | Required | Purpose |
|---|---|---|
| `queue.json` | yes | The source queue's `GetQueueAttributes` output. |
| `dlq.json` | when the queue has a DLQ | The dead-letter queue's `GetQueueAttributes` output. Needed for the R3 retention-ordering check; its ARN is the source queue's `RedrivePolicy.deadLetterTargetArn`. |

Key attributes the audit reads:

| Attribute | Type (as stored) | Used by |
|---|---|---|
| `QueueArn` | string | identification |
| `VisibilityTimeout` | string seconds | R4, R5 |
| `MessageRetentionPeriod` | string seconds | R3, R4, R6 |
| `RedrivePolicy` | JSON string (`deadLetterTargetArn`, `maxReceiveCount`) | R1, R2, R3, R4 |
| `Policy` | JSON string (IAM policy document) | R7 |
| `SqsManagedSseEnabled` / `KmsMasterKeyId` | string / string | R8 |
| `FifoQueue` / `ContentBasedDeduplication` | string booleans | R9 |

The reference implementation (`_audit.py`) accepts either the full `{"Attributes": {...}}` envelope or a bare attribute map, and treats a missing `dlq.json` as "DLQ attributes not provided" (R3 skipped).

## Adding a new replay test

When you contribute a new worked example to the skill:

1. Drop fixtures under `../fixtures/<example-slug>/` following the schema above.
2. Add `replay_NN_<slug>.py` in this directory, modeled on the existing eight. Use the shared `report` helper in `_replay.py`.
3. Assert the expected findings, the severity, and that the boundary names the relevant join.
4. Run locally, commit, and reference the test in the example's markdown narrative.

A new test that does not exercise a rule, severity, or boundary join the existing tests do not exercise will fail review. The point of the replay corpus is breadth.

## Why stdlib only

Skills get adopted when they run anywhere with zero setup. A `pip install` is an adoption tax. The reference implementation uses only `json`, `pathlib`, `dataclasses`, and `typing`. If a future test requires a third-party dependency (e.g. `boto3` or `pytest`), that's a signal the skill is leaking implementation detail: the audit operates on the `GetQueueAttributes` output, not on a live AWS connection.

## Why the reference implementation is deterministic

`_audit.py` is a deterministic stand-in for what an AI agent does when it follows `SKILL.md`. It exists so the replay tests can assert that the methodology, applied to known fixtures, produces the expected findings. A natural follow-up is to run the same fixtures through an actual LLM agent loaded with `SKILL.md` and assert it produces the same findings and names the same boundary; that work is out of scope for the first reference skill.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall