Skip to content

anyshift-io/sre-skills / skills/sqs-queue-auditor

sqs-queue-auditor

Verified

FileKindSize
FAILURE_MODES.mdmarkdown6.6 KB
README.mdmarkdown4.9 KB
SKILL.mdskill md13.3 KB
examples/01-no-dlq.mdmarkdown2.9 KB
examples/02-dlq-retention-shorter-than-source.mdmarkdown3.3 KB
examples/03-maxreceivecount-too-low.mdmarkdown2.8 KB
examples/04-poison-ages-out-before-dlq.mdmarkdown3.9 KB
examples/05-default-visibility-short-retention.mdmarkdown3.2 KB
examples/06-public-queue-policy.mdmarkdown3.6 KB
examples/07-fifo-dedup-off.mdmarkdown2.8 KB
examples/08-clean-standard.mdmarkdown2.6 KB
fixtures/01-no-dlq/queue.jsonconfig564 B
fixtures/02-dlq-retention-shorter-than-source/dlq.jsonconfig649 B
fixtures/02-dlq-retention-shorter-than-source/queue.jsonconfig632 B
fixtures/03-maxreceivecount-too-low/dlq.jsonconfig659 B
fixtures/03-maxreceivecount-too-low/queue.jsonconfig637 B
fixtures/04-poison-ages-out-before-dlq/dlq.jsonconfig663 B
fixtures/04-poison-ages-out-before-dlq/queue.jsonconfig648 B
fixtures/05-default-visibility-short-retention/dlq.jsonconfig650 B
fixtures/05-default-visibility-short-retention/queue.jsonconfig631 B
fixtures/06-public-queue-policy/dlq.jsonconfig663 B
fixtures/06-public-queue-policy/queue.jsonconfig908 B
fixtures/07-fifo-dedup-off/dlq.jsonconfig748 B
fixtures/07-fifo-dedup-off/queue.jsonconfig804 B
fixtures/08-clean-standard/dlq.jsonconfig672 B
fixtures/08-clean-standard/queue.jsonconfig1.0 KB
tests/README.mdmarkdown4.2 KB
tests/_audit.pyscript21.8 KB
tests/_replay.pyscript720 B
tests/eval/README.mdmarkdown8.4 KB
tests/eval/eval_results.jsonconfig220.4 KB
tests/eval/judge_prompt.mdmarkdown1.4 KB
tests/eval/queues.pyscript7.6 KB
tests/eval/rubric.mdmarkdown2.6 KB
tests/eval/run_eval.pyscript11.5 KB
tests/replay_01_no_dlq.pyscript1.5 KB
tests/replay_02_dlq_retention.pyscript1.5 KB
tests/replay_03_maxreceivecount.pyscript1.3 KB
tests/replay_04_poison_ages_out.pyscript1.6 KB
tests/replay_05_default_visibility.pyscript1.8 KB
tests/replay_06_public_policy.pyscript1.4 KB
tests/replay_07_fifo_dedup_off.pyscript1.4 KB
tests/replay_08_clean_standard.pyscript1.6 KB

tests/eval/rubric.md

View on GitHub
# Eval rubric: `sqs-queue-auditor`

Seven binary items (1 = pass, 0 = fail). No partial credit. The judge is given the deterministic reference audit (from `_audit.py`) as ground truth.

1. **Parse.** The agent correctly reads the queue configuration: it parses the embedded `RedrivePolicy` JSON string (identifying whether a DLQ exists and the `maxReceiveCount`), and treats the string-typed second values (`VisibilityTimeout`, `MessageRetentionPeriod`) as numbers. An agent that never opens the embedded JSON, or that misreads "no RedrivePolicy" vs "DLQ present", fails.

2. **Findings.** The agent identifies the misconfiguration(s) the ground truth lists for this fixture (by substance, not by rule code: it need not say "R4", but it must describe the same defect). For the clean control, the agent reports no defect.

3. **No false positives.** The agent does not assert a material misconfiguration that the ground truth does not contain. On the clean control this is the whole game (a wildcard principal narrowed by `aws:SourceArn` must NOT be called public). On other fixtures, inventing extra critical/high defects fails this item.

4. **Criticality.** The agent ranks severity correctly: it identifies the silent-message-loss defects (DLQ-retention-ordering, poison-ages-out) as the most serious, and does not present a low-severity flag as if it were the headline. For fixtures whose only findings are soft flags, the agent correctly treats them as lower-severity.

5. **Boundary.** The agent names at least one thing it cannot determine from the queue configuration alone, matching the ground-truth join (consumer processing time, live metrics over time, the IAM identity-policy union, or producer behaviour). An agent that presents its config audit as a complete health verdict fails.

6. **Honesty on soft flags.** Where the fixture involves a flag that depends on something outside the queue (default visibility timeout, encryption-vs-data-classification, FIFO dedup-vs-producers), the agent presents it as a flag to verify rather than asserting a confirmed bug. For fixtures with no soft flag, this item passes as long as the agent does not overclaim certainty elsewhere.

7. **Recommendation.** The agent's top recommended fix matches the ground-truth fix in substance (e.g. "raise DLQ retention to the maximum", "add an aws:SourceArn condition", "raise maxReceiveCount into a sane band").

A perfect audit scores 7. The control condition (no skill) typically loses points on items 4, 5, and 6: a cold agent finds the obvious defects but tends to present a configuration read as a full health check, misses the silent-loss arithmetic, and states soft flags with unearned certainty.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall