anyshift-io/sre-skills / skills/sqs-queue-auditor
sqs-queue-auditor
Verifiedtests/eval/eval_results.json
View on GitHub[
{
"queue": "02-dlq-retention-shorter-than-source",
"condition": "control",
"trial": 0,
"score": 6,
"scores_by_item": [
1,
1,
1,
1,
1,
0,
1
],
"notes": [
"Agent correctly parses the DLQ retention as 86400 seconds (1 day) and source as 345600 seconds (4 days), demonstrating correct string-typed attribute parsing.",
"Agent identifies the core defect: DLQ retention is shorter than source queue retention, causing messages to be silently dropped on arrival, matching the ground-truth R3 finding.",
"No false positives that mischaracterize scoped policies as public; additional findings like short polling and encryption are presented as soft flags, not critical misconfigurations.",
"Issue 1 (DLQ retention / silent message loss) is ranked as HIGH and listed first, correctly headlining the silent-loss defect above operational and compliance concerns.",
"Agent notes it cannot determine when messages arrived or confirm operational ownership, and explicitly states DLQ metrics over time require CloudWatch inspection, matching the ground-truth boundary.",
"Issues 3 (maxReceiveCount), 5 (SSE-SQS), and 6 (no resource policy) are presented as confirmed issues rather than flags to verify depending on context; for example, maxReceiveCount=5 is called potentially problematic without acknowledging it may be perfectly appropriate.",
"Top fix correctly recommends raising the DLQ MessageRetentionPeriod to 1,209,600 seconds (14-day maximum), matching the ground-truth recommendation in substance and providing the exact CLI command."
],
"agent_output": "## SQS Queue Audit: `order-events`\n\n---\n\n### Issue 1 \u2014 DLQ Retention Period Too Short\n**Severity: HIGH**\n\n**Finding:**\nThe DLQ `MessageRetentionPeriod` is `86400` seconds (24 hours). The source queue retains messages for `345600` seconds (4 days).\n\n**Risk:**\nA message that reaches the DLQ has already spent an unknown amount of time on the source queue (potentially up to 4 days). If it arrives in the DLQ near the end of that window, it can be **silently dropped within hours** before any engineer investigates it. There is no delivery failure \u2014 the message simply disappears with no further signal. For order events this means lost orders with no recovery path.\n\n**Fix:**\nSet the DLQ retention to the **maximum of 1,209,600 seconds (14 days)**. This maximises the investigation and replay window.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes MessageRetentionPeriod=1209600\n```\n\n---\n\n### Issue 2 \u2014 DLQ Has 37 Messages With No Redrive or Alarm Visible\n**Severity: HIGH**\n\n**Finding:**\n`ApproximateNumberOfMessages` on the DLQ is `37`. There is no `RedriveAllowPolicy` configured on the source queue to permit redrive back, and no evidence of active consumption or alerting on this queue.\n\n**Risk:**\n37 order events have already failed processing and are sitting unhandled. Given Issue 1 above, they are also **actively expiring**. If these represent real orders they are unprocessed business events. The queue depth also indicates the worker failure mode is live and recurring, not historical.\n\n**Fix:**\nImmediate actions:\n1. Determine when these messages arrived (CloudWatch `NumberOfMessagesSent` on the DLQ).\n2. Investigate worker errors in logs before any redrive.\n3. Once root cause is confirmed, redrive using the SQS console or:\n\n```bash\naws sqs start-message-move-task \\\n --source-arn arn:aws:sqs:eu-west-1:211125758836:order-events-dlq \\\n --destination-arn arn:aws:sqs:eu-west-1:211125758836:order-events\n```\n\n4. Create a CloudWatch alarm on `ApproximateNumberOfMessagesVisible` for the DLQ with a threshold of `0` (alert on any message).\n\n---\n\n### Issue 3 \u2014 `maxReceiveCount` of 5 May Be Too Low Depending on Worker Behaviour\n**Severity: MEDIUM**\n\n**Finding:**\n`maxReceiveCount` is `5`. This means a message is moved to the DLQ after 5 failed receive-and-not-delete cycles.\n\n**Risk:**\nIf workers experience transient infrastructure failures (network blips, cold starts, deployment rollouts), 5 attempts may be insufficient to distinguish a genuinely poisoned message from a temporarily unhealthy worker. This causes **unnecessary DLQ promotion** of recoverable messages, which then require manual redrive and carry the retention risk from Issue 1.\n\nConversely, if processing has side effects (e.g. partial order writes), each retry increases the risk of **partial duplication** unless the worker is fully idempotent.\n\n**Fix:**\n- If workers are idempotent and transient failures are common: raise to `10`.\n- If workers are not fully idempotent: keep low but **ensure idempotency** at the application layer (deduplicate on order ID) before raising it.\n- Document the chosen value and the rationale.\n\n---\n\n### Issue 4 \u2014 DLQ `ReceiveMessageWaitTimeSeconds` is `0` (Short Polling)\n**Severity: LOW**\n\n**Finding:**\nThe source queue correctly uses long polling (`ReceiveMessageWaitTimeSeconds=20`). The DLQ is set to `0`, meaning short polling.\n\n**Risk:**\nThe DLQ is likely polled by an alerting consumer, ops tooling, or a redrive process. Short polling can return **empty responses even when messages exist** (it samples a subset of SQS servers), leading to missed detections during incident triage and unnecessary API cost.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n### Issue 5 \u2014 Encryption Uses SSE-SQS, Not SSE-KMS\n**Severity: LOW (Compliance-Dependent)**\n\n**Finding:**\nBoth queues have `SqsManagedSseEnabled=true`, meaning AWS manages the encryption key. There is no customer-managed KMS key (CMK).\n\n**Risk:**\n- You have **no control over key rotation policy**, no ability to revoke access via key policy, and no CloudTrail visibility into individual decrypt operations.\n- For order events containing PII or payment-adjacent data this is likely a compliance gap (PCI-DSS, GDPR depending on message content).\n- A compromised AWS account has no key-level blast radius control.\n\n**Fix:**\nIf regulatory or security requirements demand it, migrate to SSE-KMS with a CMK:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n --attributes KmsMasterKeyId=<your-cmk-arn>\n```\n\nEnsure the worker fleet IAM roles and the DLQ both reference the same CMK with appropriate `kms:Decrypt` / `kms:GenerateDataKey` grants.\n\n---\n\n### Issue 6 \u2014 No SQS Resource Policy Present\n**Severity: LOW (Defence-in-Depth)**\n\n**Finding:**\nNeither queue has a resource-based policy (`Policy` attribute) returned by the API, which means access is controlled **solely by IAM identity policies**.\n\n**Risk:**\n- There is no explicit deny to prevent other principals in account `211125758836` from reading or purging these queues if their IAM policies are overly broad.\n- No cross-account access is scoped, so any future cross-account IAM grants would have no queue-side guardrail.\n\n**Fix:**\nAdd a least-privilege resource policy that explicitly allows only the expected worker role ARNs and denies all other principals:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowWorkerFleet\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": \"arn:aws:iam::211125758836:role/order-worker-role\"\n },\n \"Action\": [\n \"sqs:ReceiveMessage\",\n \"sqs:DeleteMessage\",\n \"sqs:ChangeMessageVisibility\",\n \"sqs:GetQueueAttributes\"\n ],\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:order-events\"\n },\n {\n \"Sid\": \"DenyEverythingElse\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"sqs:*\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:order-events\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"aws:PrincipalArn\": \"arn:aws:iam::211125758836:role/order-worker-role\"\n }\n }\n }\n ]\n}\n```\n\n---\n\n### Summary Table\n\n| # | Issue | Severity | Message Loss Risk |\n|---|-------|----------|-------------------|\n| 1 | DLQ retention 24h vs 4-day source queue | **HIGH** | Direct \u2014 messages expire |\n| 2 | 37 unhandled DLQ messages, no alerting | **HIGH** | Active \u2014 orders already lost |\n| 3 | `maxReceiveCount=5` may be too aggressive | **MEDIUM** | Indirect \u2014 unnecessary DLQ promotion |\n| 4 | DLQ short polling | **LOW** | Operational \u2014 missed triage signal |\n| 5 | SSE-SQS not SSE-KMS | **LOW** | Compliance/exposure risk |\n| 6 | No resource policy | **LOW** | Exposure risk |\n\n**Immediate action required on Issues 1 and 2.** The DLQ messages are expiring now.",
"elapsed_s": 55.7
},
{
"queue": "02-dlq-retention-shorter-than-source",
"condition": "control",
"trial": 1,
"score": 5,
"scores_by_item": [
1,
1,
1,
1,
0,
0,
1
],
"notes": [
"Agent correctly parses the 345600s (4-day) source retention and 86400s (1-day) DLQ retention, identifying the numeric mismatch accurately.",
"Agent identifies the core R3 defect \u2014 DLQ retention shorter than source retention causing silent message loss due to SentTimestamp not resetting on redrive.",
"Agent does not flag any false positives; no access policies are miscalled as public or other nonexistent misconfigurations asserted.",
"Agent correctly ranks the DLQ retention issue as Critical and leads with it as the most serious finding.",
"Agent does not explicitly name any boundary it cannot cross (e.g., cannot see DLQ metrics over time or identify the operational owner); the boundary acknowledgment is absent.",
"Several findings (maxReceiveCount, VisibilityTimeout, KMS) are presented as confirmed bugs or definite problems rather than flags to verify depending on workload characteristics.",
"The top fix correctly recommends raising DLQ MessageRetentionPeriod to the 14-day maximum (1209600 seconds), matching the ground-truth recommendation in substance."
],
"agent_output": "## SQS Queue Audit: `order-events`\n\n---\n\n### CRITICAL\n\n#### 1. DLQ retention period shorter than source queue retention period\n\n**What's configured:**\n- Source queue `MessageRetentionPeriod`: `345600` (4 days)\n- DLQ `MessageRetentionPeriod`: `86400` (1 day)\n\n**Why this is a problem:**\nA message can spend up to 4 days being retried on the source queue before it lands in the DLQ. Once it arrives, it has only 1 day left \u2014 but the DLQ clock uses the **original enqueue timestamp**, not the arrival timestamp at the DLQ. A message that exhausts retries on day 3 arrives in the DLQ with roughly 1 day of retention remaining. A message that exhausts retries on day 4 may expire **immediately on arrival** or already be gone. You will silently lose failed messages with no chance to inspect or replay them.\n\n**Fix:**\nSet the DLQ retention to **at least as long as the source queue**, ideally longer to give operators time to investigate. The maximum is 14 days.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes MessageRetentionPeriod=1209600 # 14 days\n```\n\n---\n\n### HIGH\n\n#### 2. DLQ has 37 messages with no redrive alarm or apparent consumption\n\n**What's configured:**\n- `ApproximateNumberOfMessages`: `37` on the DLQ\n- No redrive policy on the DLQ (no DLQ-of-DLQ configured)\n- `RedriveAllowPolicy` is set correctly, but there is no evidence of a consumer or alarm\n\n**Why this is a problem:**\n37 messages sitting in the DLQ means 37 order events have already failed all 5 retry attempts. Given the 1-day retention window (see issue #1), these messages are at active risk of expiry. Even under a corrected retention period, a DLQ without alerting is operationally silent \u2014 failures accumulate unnoticed until business impact occurs.\n\n**Fix:**\n- Immediately inspect and replay or action the 37 existing messages before they expire.\n- Create a CloudWatch alarm on `ApproximateNumberOfMessagesVisible` for the DLQ:\n\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"order-events-dlq-not-empty\" \\\n --metric-name ApproximateNumberOfMessagesVisible \\\n --namespace AWS/SQS \\\n --dimensions Name=QueueName,Value=order-events-dlq \\\n --statistic Sum \\\n --period 60 \\\n --threshold 1 \\\n --comparison-operator GreaterThanOrEqualToThreshold \\\n --evaluation-periods 1 \\\n --alarm-actions <your-sns-topic-arn>\n```\n\n---\n\n### MEDIUM\n\n#### 3. `maxReceiveCount` of 5 may be too low depending on worker failure modes\n\n**What's configured:**\n- `maxReceiveCount`: `5`\n- `VisibilityTimeout`: `60` seconds\n\n**Why this is a problem:**\nWith only 5 receive attempts, a message that hits transient infrastructure failures (cold starts, downstream timeouts, brief DB unavailability) will be moved to the DLQ quickly. 5 attempts at 60-second visibility timeout means a message can be dead-lettered in as little as 5 minutes. This is appropriate for poison-pill detection, but too aggressive if your workers experience legitimate transient errors lasting more than a few minutes. The current DLQ depth of 37 may partly reflect this.\n\n**Fix:**\nEvaluate your worker's error profile. If transient failures are common, increase `maxReceiveCount` to `10`\u2013`20`. If you need faster poison-pill isolation, keep it low but ensure your worker distinguishes transient vs. permanent errors and uses `ChangeMessageVisibility` to back off rather than immediately releasing the message.\n\n---\n\n### MEDIUM\n\n#### 4. VisibilityTimeout may not account for worst-case processing time\n\n**What's configured:**\n- `VisibilityTimeout`: `60` seconds on both queues\n\n**Why this is a problem:**\nIf any worker takes longer than 60 seconds to process an order event (e.g., downstream API latency, large payload processing), the message becomes visible again and a second worker will pick it up. This causes **duplicate processing**. Both workers will likely succeed and delete the message \u2014 but only one delete will succeed; the other is a no-op \u2014 or, worse, both complete business logic (e.g., charge a customer, create an order) before either deletes the message. The `VisibilityTimeout` must be a safe upper bound for processing time, not a typical-case estimate.\n\n**Fix:**\nSet the visibility timeout to your **P99 processing time plus a safety margin** (at least 1.5\u00d7). If your workers are long-running, implement heartbeating by calling `ChangeMessageVisibility` periodically to extend the timeout during processing. If P99 is genuinely under 60 seconds, document that explicitly so future changes don't regress it silently.\n\n---\n\n### LOW\n\n#### 5. DLQ `ReceiveMessageWaitTimeSeconds` is 0 (short polling)\n\n**What's configured:**\n- Source queue: `ReceiveMessageWaitTimeSeconds`: `20` (long polling \u2014 correct)\n- DLQ: `ReceiveMessageWaitTimeSeconds`: `0` (short polling)\n\n**Why this is a problem:**\nIf you have an automated consumer or replay tooling that polls the DLQ, short polling will increase API call costs and may miss messages distributed across SQS internal partitions on any given poll. This is low severity because the DLQ typically has low-frequency manual or automated replay traffic, but it is an inconsistency that can cause incomplete reads.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n### INFORMATIONAL\n\n#### 6. Encryption is SQS-managed, not KMS-managed\n\n**What's configured:**\n- `SqsManagedSseEnabled`: `true` on both queues\n- No `KmsMasterKeyId` attribute present\n\n**Why this is a problem:**\nSQS-managed SSE encrypts at rest but gives you no key rotation control, no ability to revoke access via key policy, no CloudTrail visibility of decrypt operations, and no cross-account key sharing capability. For order events containing customer or payment-adjacent data, this is likely below your compliance bar.\n\n**Fix:**\nCreate a customer-managed KMS key with an appropriate key policy and migrate both queues:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n --attributes KmsMasterKeyId=<your-cmk-arn>\n```\n\nRepeat for the DLQ. Ensure your worker IAM roles have `kms:Decrypt` and `kms:GenerateDataKey` on the key.\n\n---\n\n### Summary Table\n\n| # | Issue | Severity | Risk |\n|---|-------|----------|------|\n| 1 | DLQ retention (1d) < source retention (4d) | **Critical** | Silent message loss |\n| 2 | 37 messages in DLQ, no alarm, expiring | **High** | Active data loss |\n| 3 | `maxReceiveCount=5` may be too aggressive | **Medium** | Premature DLQ routing |\n| 4 | VisibilityTimeout may allow duplicate processing | **Medium** | Duplicate order processing |\n| 5 | DLQ using short polling | **Low** | Cost, incomplete reads |\n| 6 | SQS-managed SSE instead of KMS CMK | **Informational** | Compliance/audit gap |",
"elapsed_s": 46.8
},
{
"queue": "02-dlq-retention-shorter-than-source",
"condition": "control",
"trial": 2,
"score": 5,
"scores_by_item": [
1,
1,
1,
1,
0,
0,
1
],
"notes": [
"Correctly parses RedrivePolicy and identifies 86400s (1 day) DLQ vs 345600s (4 days) source retention.",
"Identifies the ground-truth R3 defect: DLQ retention shorter than source, with correct SentTimestamp reasoning.",
"No false positives introduced; all findings are plausible operational concerns, not fabricated misconfigs.",
"Ranks the DLQ retention issue as CRITICAL and leads with it, correctly prioritizing silent message loss.",
"Agent never names a specific thing it cannot determine from config alone (e.g., how many messages were already silently dropped, or who the operational owner is); boundary condition is not acknowledged.",
"Several soft flags (maxReceiveCount, SSE-SQS vs CMK, DLQ-on-DLQ) are presented as confirmed issues rather than 'flags to verify depending on context'; for example, maxReceiveCount:5 is called a risk without acknowledging it may be deliberately chosen.",
"Top fix correctly recommends raising DLQ MessageRetentionPeriod to 1209600 (14-day maximum), matching ground truth."
],
"agent_output": "## SQS Queue Audit: `order-events`\n\n---\n\n### CRITICAL\n\n#### 1. DLQ Retention Period Too Short (Message Loss)\n\n**Finding:** `order-events-dlq` has `MessageRetentionPeriod: 86400` (24 hours). The source queue retains messages for `345600` seconds (4 days).\n\n**Impact:** A message can spend up to 4 days being retried on the source queue before landing in the DLQ. Once it arrives in the DLQ it could be close to or already past the 24-hour DLQ window, meaning it gets **silently dropped before anyone can investigate or replay it**. Even in normal failure scenarios, 24 hours gives very little time for an on-call engineer to be paged, investigate, and act before evidence is gone.\n\n**Fix:** Set DLQ retention to at least as long as the source queue, preferably the SQS maximum:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes MessageRetentionPeriod=1209600 # 14 days\n```\n\n---\n\n### HIGH\n\n#### 2. No DLQ Alarm on Depth (Silent Failures)\n\n**Finding:** `ApproximateNumberOfMessages: 37` on the DLQ right now, with no CloudWatch alarm visible in this configuration. 37 messages are already dead.\n\n**Impact:** Failed messages accumulate silently. The DLQ is only useful if someone is alerted to drain or replay it. Without an alarm this is operationally equivalent to having no DLQ \u2014 failures are invisible until someone manually checks.\n\n**Fix:** Create an alarm that fires as soon as any message lands in the DLQ:\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"order-events-dlq-not-empty\" \\\n --metric-name ApproximateNumberOfMessagesVisible \\\n --namespace AWS/SQS \\\n --dimensions Name=QueueName,Value=order-events-dlq \\\n --statistic Sum \\\n --period 60 \\\n --threshold 1 \\\n --comparison-operator GreaterThanOrEqualToThreshold \\\n --evaluation-periods 1 \\\n --alarm-actions <your-sns-topic-arn>\n```\n\n---\n\n#### 3. `maxReceiveCount: 5` May Be Too Low or Too High Depending on Worker Behaviour (Message Loss / Incorrect DLQ Routing)\n\n**Finding:** A message is moved to the DLQ after 5 receive attempts. This interacts directly with `VisibilityTimeout: 60`.\n\n**Two concrete risks:**\n\n- **Too low for transient errors:** If your workers have retry/backoff logic within a single receive (re-queuing via visibility timeout extension), 5 attempts is exhausted quickly during a downstream outage, sending legitimately retriable messages to the DLQ prematurely.\n- **Too high for poison pills:** If the message is a hard poison pill (malformed payload causing worker crash), 5 attempts means 5 worker failures and 5 \u00d7 60s = potentially several minutes of blocked concurrency per message before quarantine, depending on fleet size.\n\n**Fix:** Audit worker retry behaviour. A common safe default for order processing is `maxReceiveCount: 3` for fast-fail on poison pills, combined with explicit worker-side retry logic with exponential backoff for transient errors. Align the value deliberately rather than leaving it at the default. Update via:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\\\",\\\"maxReceiveCount\\\":3}\"}'\n```\n\n---\n\n### MEDIUM\n\n#### 4. DLQ `ReceiveMessageWaitTimeSeconds: 0` (Unnecessary Cost / Polling Behaviour)\n\n**Finding:** The DLQ uses short polling (`ReceiveMessageWaitTimeSeconds: 0`) while the source queue correctly uses long polling (`20`).\n\n**Impact:** If any automated consumer or replay tooling polls the DLQ (e.g. a Lambda with event source mapping, or a manual replay script), short polling returns empty responses frequently, increases API call costs, and can miss messages present in non-sampled partitions. Not a message-loss risk in isolation, but operationally sloppy and can cause replay tooling to appear to report an empty DLQ when it is not.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n#### 5. Source Queue Retention Period Is Only 4 Days (Marginal for Order Events)\n\n**Finding:** `MessageRetentionPeriod: 345600` = 4 days on the source queue.\n\n**Impact:** If the worker fleet goes completely down (deployment failure, IAM breakage, capacity event) for more than 4 days, unprocessed order events are **permanently dropped with no recovery path**. For financial/order data, 4 days is a tight margin.\n\n**Fix:** Evaluate business requirements. For order events, consider raising to 7\u201314 days (SQS max is 14 days = 1209600 seconds) to give adequate recovery runway:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n --attributes MessageRetentionPeriod=1209600\n```\n\n---\n\n#### 6. Encryption Is SQS-Managed (SSE-SQS), Not KMS\n\n**Finding:** `SqsManagedSseEnabled: true` on both queues. No `KmsMasterKeyId` is present.\n\n**Impact:** SSE-SQS encrypts at rest, but the key is AWS-managed with no customer control. You cannot:\n- Audit key usage via CloudTrail\n- Restrict queue access by key policy\n- Rotate or revoke the key independently\n- Meet compliance requirements (PCI-DSS, SOC2 controls) that mandate customer-managed keys for sensitive data\n\nOrder events likely contain PII and financial data.\n\n**Fix:** Switch to a CMK:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n --attributes KmsMasterKeyId=<your-cmk-arn>\n```\nApply the same to the DLQ. Ensure your worker IAM roles have `kms:Decrypt` and `kms:GenerateDataKey` on the CMK.\n\n---\n\n### LOW\n\n#### 7. No Dead-Letter Queue on the DLQ Itself\n\n**Finding:** The DLQ has no `RedrivePolicy` of its own.\n\n**Impact:** If a consumer of the DLQ (e.g. a replay Lambda) itself fails repeatedly while processing DLQ messages, SQS will eventually make those messages invisible and ultimately drop them when retention expires. This is low severity because DLQ consumers should be simple, but it is a gap in the safety net.\n\n**Fix:** Attach a secondary DLQ (`order-events-dlq-dlq`) to the DLQ with a low `maxReceiveCount` (e.g. 3) and a maximum retention period. Alert on any messages reaching it.\n\n---\n\n### Summary Table\n\n| # | Issue | Severity | Risk |\n|---|-------|----------|------|\n| 1 | DLQ retention 24h < source queue 4d | **CRITICAL** | Silent message loss |\n| 2 | No alarm on DLQ depth (37 msgs dead now) | **HIGH** | Silent failures |\n| 3 | `maxReceiveCount:5` not deliberately set | **HIGH** | Premature DLQ or slow poison-pill quarantine |\n| 4 | DLQ short polling | **MEDIUM** | Replay tooling unreliable, cost |\n| 5 | Source queue retention only 4 days | **MEDIUM** | Message loss on extended outage |\n| 6 | SSE-SQS not CMK | **MEDIUM** | Compliance gap, no key audit trail |\n| 7 | No DLQ-on-DLQ | **LOW** | Gap in safety net |",
"elapsed_s": 47.0
},
{
"queue": "02-dlq-retention-shorter-than-source",
"condition": "treatment",
"trial": 0,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON and converts all string-typed seconds to human-readable durations.",
"Identifies R3 (DLQ retention 1 day < source retention 4 days) with correct SentTimestamp reasoning, matching the ground-truth defect.",
"No false positives; extra rules R8/R9 etc. are evaluated and correctly dismissed.",
"R3 is ranked CRITICAL and headlined as the primary finding; no soft flag is elevated above it.",
"Explicitly names inability to determine dropped-message counts over time (CloudWatch metrics join) and operational owner/consumer, matching ground-truth boundary.",
"Consumer processing time, live redrive behaviour, and IAM effective permissions are presented as items to verify, not confirmed bugs.",
"Top fix recommends raising DLQ MessageRetentionPeriod above source retention, ideally toward the maximum, matching the ground-truth fix."
],
"agent_output": "# SQS Queue Audit \u2014 `order-events`\n\n---\n\n## Step 1: Parse the Attributes\n\n**Source queue (`order-events`)**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:order-events` | \u2014 |\n| `VisibilityTimeout` | `\"60\"` | 60 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `{\"deadLetterTargetArn\":\"arn:\u2026:order-events-dlq\",\"maxReceiveCount\":5}` | DLQ wired; maxReceiveCount = 5 |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**Dead-letter queue (`order-events-dlq`)**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` | \u2014 |\n| `MessageRetentionPeriod` | `\"86400\"` | 86,400 s (1 day) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n\n---\n\n## Step 2\u20135: Rule Evaluation\n\n### R1 \u2014 No DLQ\n`RedrivePolicy` is present, pointing to `order-events-dlq`. **Not fired.**\n\n### R2 \u2014 maxReceiveCount outside 3\u201310\n`maxReceiveCount = 5`. Within the sane band [3, 10]. **Not fired.**\n\n### R3 \u2014 DLQ retention not longer than source *(critical)*\n- Source retention: **345,600 s (4 days)**\n- DLQ retention: **86,400 s (1 day)**\n- DLQ retention (86,400 s) is **less than** source retention (345,600 s). **Fired.**\n\nA message's `SentTimestamp` is set at first enqueue and is never reset when SQS moves the message to the DLQ. A message that is processed slowly \u2014 held in the source queue for, say, 70 hours before its receive count is exhausted \u2014 arrives in the DLQ with 70 hours of age already counted against a 24-hour retention window. It is deleted from the DLQ almost immediately, with no time for an engineer to inspect or replay it. The DLQ reads as wired and healthy; the messages you most need to examine are the ones it silently discards.\n\n### R4 \u2014 Poison messages age out before reaching the DLQ *(critical)*\n- Minimum wall-clock to exhaust receive count: `maxReceiveCount \u00d7 VisibilityTimeout = 5 \u00d7 60 = 300 s`\n- Source retention: `345,600 s`\n- `300 s \u2264 345,600 s` \u2192 inequality does **not** hold. **Not fired.**\n\nThe arithmetic is satisfactory: a poison message has more than enough retention headroom to exhaust all five delivery attempts and reach the DLQ under the configured values.\n\n### R5 \u2014 Visibility timeout at 30 s default\n`VisibilityTimeout = 60 s`. Not at the 30 s default. **Not fired.**\n\n### R6 \u2014 Retention shorter than a plausible outage *(medium)*\n- Source retention: **345,600 s (4 days)**. Well above the 1-hour threshold. **Not fired.**\n- DLQ retention: **86,400 s (1 day)**. Above the 1-hour threshold; however, R3 above already captures the more precise and severe form of this problem.\n\n### R7 \u2014 Open resource policy\nNo `Policy` attribute on either queue. **Not fired.**\n\n### R8 \u2014 Encryption at rest disabled\n`SqsManagedSseEnabled = \"true\"` on both queues. **Not fired.**\n\n### R9 \u2014 FIFO with content-based dedup off\nStandard queue; FIFO invariants do not apply. **Not fired.**\n\n---\n\n## Output\n\n### 1. Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| **Type** | Standard |\n| **DLQ** | Wired \u2192 `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n\n---\n\n### 2. Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R3: DLQ retention is shorter than source retention\n\n**Attributes:**\n- Source `MessageRetentionPeriod`: 345,600 s (4 days)\n- DLQ `MessageRetentionPeriod`: 86,400 s (1 day)\n\n**What breaks:**\nSQS preserves the original `SentTimestamp` when a message is moved to the DLQ \u2014 it does not reset the clock. A message that spends most of its source-queue lifetime being retried (e.g., 80+ hours before its fifth delivery attempt fails) arrives in the DLQ with the majority of the DLQ's 24-hour window already consumed. In the worst case it arrives already expired and is deleted on ingestion. The DLQ is non-empty (37 messages are visible right now), which confirms it is receiving messages \u2014 but those 37 messages are the *fast* failures. Slow failures \u2014 the ones most likely to reflect a systemic processing problem \u2014 may be aging out of the DLQ invisibly. There is no error surfaced anywhere; from the console the queue looks correct.\n\n**Recommendation:**\nSet the DLQ's `MessageRetentionPeriod` to **strictly greater than** the source's. A practical rule: DLQ retention \u2265 source retention + one operational response window. Given a 4-day source, a DLQ retention of **7 days** (604,800 s) is a reasonable baseline and SQS's own recommendation for DLQ configurations. Update via:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes MessageRetentionPeriod=604800\n```\n\nOr in Terraform/CDK, set `message_retention_seconds = 604800` on the DLQ resource and ensure the value is parameterised relative to the source so the relationship is maintained if the source retention is ever changed.\n\n---\n\n*No further findings. All other rules evaluated clean against the configured values.*\n\n---\n\n### 3. Boundary\n\nThe following questions this audit cannot answer from queue configuration alone. They are not omissions; they are the joins across resources, time, and account that a single `GetQueueAttributes` read cannot provide. A clean answer to R3 does not close them.\n\n**Consumer processing time vs. visibility timeout (R5 territory)**\nThe visibility timeout is 60 s. Whether that is sufficient depends entirely on how long the worker fleet takes to process an order event. If any processing path \u2014 database lookups, downstream HTTP calls, enrichment logic \u2014 routinely exceeds 60 s, the message reappears mid-processing and is delivered to a second worker concurrently. This audit cannot see consumer processing time. Check your CloudWatch `ApproximateAgeOfOldestMessage` and worker execution duration metrics, and extend `VisibilityTimeout` or use `ChangeMessageVisibility` heartbeating if there is any margin.\n\n**Live redrive behaviour**\nThere are currently 37 messages in the DLQ and 88 in the source queue. The audit cannot determine whether the DLQ messages are accumulating (a processing regression), are a stable residue from a past incident, or are being drained. The R3 finding means some messages from the source queue may already have been silently deleted by the DLQ before they could be counted. CloudWatch `NumberOfMessagesSent` and `NumberOfMessagesDeleted` on the DLQ over time is the only way to quantify the actual loss.\n\n**IAM effective permissions**\nThere is no resource policy on `order-events`. The effective set of principals that can send to, receive from, purge, or delete the queue is determined entirely by identity-based IAM policies in account `211125758836`. This audit cannot read those. The access story is half-told: no resource policy does not mean restricted access, it means all access control is in identity policies that are not visible here.\n\n**DLQ consumer and replay path**\nThe DLQ has a `RedriveAllowPolicy` scoped to the source queue ARN, which is correct. But whether anyone is actually consuming the DLQ \u2014 alerting on it, triaging messages, and replaying valid ones \u2014 is a property of the operational setup, not the configuration. A DLQ that fills silently and is never drained provides no recovery guarantee. Confirm there is an alarm on `ApproximateNumberOfMessagesVisible > 0` for `order-events-dlq` and a documented runbook for replay.\n\n**Producer contract**\nWhether the right services are writing to `order-events`, at the expected rate, with the expected message shape, is not visible in queue attributes. A misconfigured producer that stops sending is indistinguishable from a quiet queue at the configuration layer.",
"elapsed_s": 67.7
},
{
"queue": "02-dlq-retention-shorter-than-source",
"condition": "treatment",
"trial": 1,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses the JSON-encoded RedrivePolicy (maxReceiveCount=5, DLQ ARN) and converts all string-typed seconds to human-readable durations.",
"Correctly identifies R3 as the sole finding: DLQ retention (1 day) shorter than source retention (4 days), causing silent message loss on arrival.",
"No false positives; all other rules are correctly evaluated as passing, including the absent resource policy and SSE state.",
"R3 is ranked CRITICAL and headlined first, correctly prioritizing the silent message-loss defect above all other checks.",
"Explicitly names boundaries the audit cannot cross, including DLQ operational behavior over time and IAM-based access control, matching the ground-truth join.",
"Consumer processing time and producer behavior are presented as items to verify rather than confirmed bugs.",
"Top fix is to raise the DLQ MessageRetentionPeriod above the source's (recommends 7 days vs 4-day source), matching the ground-truth recommendation in substance."
],
"agent_output": "# SQS Queue Audit: order-events\n\n---\n\n## Step 1 \u2014 Parse the attributes\n\n**Source queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| `VisibilityTimeout` | `\"60\"` | 60 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345 600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE on |\n| `RedrivePolicy` | `\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\\\",\\\"maxReceiveCount\\\":5}\"` | DLQ wired; `maxReceiveCount` = 5 |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**Dead-letter queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n| `MessageRetentionPeriod` | `\"86400\"` | 86 400 s (1 day) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE on |\n\n---\n\n## Step 2 \u2014 Work the arithmetic before judgment\n\n**R3 check \u2014 DLQ retention vs source retention**\n\n```\nSource retention : 345 600 s (4 days)\nDLQ retention : 86 400 s (1 day)\n86 400 < 345 600 \u2192 DLQ retention is SHORTER than source retention \u2192 R3 FIRES\n```\n\n**R4 check \u2014 poison messages aging out before reaching the DLQ**\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout = 5 \u00d7 60 = 300 s\nMessageRetentionPeriod = 345 600 s\n300 < 345 600 \u2192 inequality NOT satisfied \u2192 R4 does NOT fire\n```\n\n**R2 check \u2014 maxReceiveCount band (3\u201310)**\n\n```\nmaxReceiveCount = 5 \u2192 within [3, 10] \u2192 R2 does NOT fire\n```\n\n**R5 check \u2014 visibility timeout at the 30 s default**\n\n```\nVisibilityTimeout = 60 s \u2260 30 s \u2192 R5 does NOT fire\n```\n\n**R6 check \u2014 retention shorter than a plausible outage**\n\n```\nSource retention = 345 600 s (4 days) > 3 600 s (1 hour) \u2192 R6 does NOT fire\n```\n\n**R7 check \u2014 open resource policy**\n\n```\nPolicy = absent \u2192 R7 does NOT fire\n```\n\n**R8 check \u2014 encryption at rest**\n\n```\nSqsManagedSseEnabled = true (source)\nSqsManagedSseEnabled = true (DLQ)\n\u2192 R8 does NOT fire\n```\n\n**R9 check \u2014 FIFO dedup**\n\n```\nFifoQueue = absent (standard queue) \u2192 R9 does NOT fire\n```\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2192 `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n| **maxReceiveCount** | 5 |\n| **Source retention** | 345 600 s (4 days) |\n| **DLQ retention** | 86 400 s (1 day) |\n| **Visibility timeout** | 60 s |\n| **Encryption** | SQS-managed SSE, both queues |\n\n---\n\n### Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R3: DLQ retention is shorter than source retention\n\n**Rule:** R3 \n**Attributes:**\n- Source `MessageRetentionPeriod`: `345600` (4 days)\n- DLQ `MessageRetentionPeriod`: `86400` (1 day)\n\n**What breaks:**\n\nSQS measures a message's age from its original `SentTimestamp` and does **not** reset that timestamp when the message is moved to the DLQ. A message that is processed and fails repeatedly across several days can arrive in the DLQ already carrying most or all of its 4-day age. With the DLQ set to only 1 day of retention, any message that took longer than 1 day to exhaust its receive count arrives in the DLQ and is deleted almost immediately \u2014 often within seconds. The DLQ will appear correctly wired, non-empty at a glance, and operationally silent about the loss.\n\nThis is the worst-case scenario for this class of bug: it is invisible in the console, it affects exactly the messages that were hardest to process (the ones that failed repeatedly over an extended period), and those are precisely the messages an on-call engineer most needs to inspect during an incident.\n\nWith 37 messages currently sitting in the DLQ (`ApproximateNumberOfMessages: 37`), it is worth checking immediately whether any of those messages are close to or already past their age limit.\n\n**Recommendation:**\n\nSet the DLQ's `MessageRetentionPeriod` to **strictly greater than** the source's retention. The standard pattern is source retention plus a meaningful investigation buffer. With a 4-day source:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes MessageRetentionPeriod=604800 # 7 days\n```\n\nA 7-day DLQ retention against a 4-day source gives a 3-day investigation window even for a message that failed on its last possible attempt at the very end of the source window. In Terraform:\n\n```hcl\nresource \"aws_sqs_queue\" \"order_events_dlq\" {\n name = \"order-events-dlq\"\n message_retention_seconds = 604800 # 7 days; must exceed source (345600 / 4 days)\n}\n```\n\nTreat the rule as a **hard invariant**: DLQ retention must always be set to a value strictly greater than source retention, and the two values must be reviewed together whenever either is changed.\n\n---\n\n### No further findings\n\nAll other rules pass cleanly against the static configuration:\n\n| Rule | Result | Notes |\n|---|---|---|\n| R1 \u2014 No DLQ | \u2705 Pass | DLQ is wired |\n| R2 \u2014 maxReceiveCount band | \u2705 Pass | 5 is within [3, 10] |\n| R4 \u2014 Poison ages out before DLQ | \u2705 Pass | 5 \u00d7 60 s = 300 s \u226a 345 600 s |\n| R5 \u2014 Default visibility timeout | \u2705 Pass | 60 s, not the 30 s default |\n| R6 \u2014 Short retention | \u2705 Pass | 4-day source retention |\n| R7 \u2014 Open resource policy | \u2705 Pass | No resource policy present |\n| R8 \u2014 Encryption disabled | \u2705 Pass | SSE enabled on both queues |\n| R9 \u2014 FIFO dedup off | \u2705 N/A | Standard queue |\n\n---\n\n### Boundary\n\nThis audit is complete and correct for what the static configuration of one queue can tell you. The following questions cannot be answered from this configuration alone and remain open.\n\n**1. Consumer processing time vs visibility timeout**\n\nThe 60-second visibility timeout is not at the 30-second default, so R5 did not fire. Whether 60 seconds is actually sufficient depends entirely on how long the worker fleet takes to process a single order event. If any worker takes longer than 60 seconds \u2014 including under load, during cold starts, or when a downstream dependency is slow \u2014 the message reappears in the queue and is delivered to a second worker while the first is still processing it. That produces duplicate processing, not message loss, but it is the same class of silent operational problem. This cannot be determined from queue attributes; it requires measurement of consumer processing time or inspection of consumer code.\n\n**2. Live queue behaviour**\n\nThe audit is against static configuration, not behaviour. The following CloudWatch metrics are not visible here and may indicate problems a clean config cannot reveal: `ApproximateAgeOfOldestMessage` (if rising, messages are not being drained fast enough), `NumberOfMessagesMoved` (actual DLQ redrive volume), `NumberOfEmptyReceives` (whether consumers are polling an idle queue at cost). The 88 messages currently in the source queue and 37 in the DLQ are point-in-time snapshots, not trends.\n\n**3. The effective access-control boundary**\n\nNo resource policy is attached to this queue, which means access is governed entirely by IAM identity policies in the account. This audit cannot read those. Any IAM role or user in account `211125758836` with `sqs:SendMessage`, `sqs:ReceiveMessage`, or `sqs:DeleteMessage` on this queue's ARN (or on `*`) has access, and none of that is visible here. The effective principal set is the union of the resource policy (empty) and every identity policy in the account \u2014 only half of which is present in this audit.\n\n**4. Producer behaviour and message contracts**\n\nWhether the right services are writing to `order-events`, whether message bodies conform to the schema the worker fleet expects, and whether any producer is sending poison messages (malformed payloads that will always fail deserialization) is not a queue attribute. A clean redrive config and a correctly-sized DLQ do not prevent a producer sending messages that will always dead-letter.\n\n**5. DLQ drainage and alerting**\n\nThe DLQ currently holds 37 messages. Whether anyone is actively monitoring it, alerting on a non-zero `ApproximateNumberOfMessages`, or draining it through a redrive-to-source process is not captured in queue configuration. A DLQ that fills silently and is never inspected provides no operational value. Confirm that a CloudWatch alarm on `ApproximateNumberOfMessages > 0` is attached to this DLQ and routed to an actionable alert channel.",
"elapsed_s": 56.9
},
{
"queue": "02-dlq-retention-shorter-than-source",
"condition": "treatment",
"trial": 2,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses the embedded RedrivePolicy JSON, maxReceiveCount=5, and all string-typed seconds into human-readable durations.",
"Identifies R3 as the sole critical finding: DLQ retention (1 day) is shorter than source retention (4 days), with the correct SentTimestamp-not-reset reasoning.",
"No false positives; all other rules are correctly marked as passing with accurate reasoning.",
"R3 is headlined as critical and framed as silent message loss, matching the ground-truth severity ranking.",
"Explicitly states it cannot determine metrics over time (dropped messages, backlog trends) and cannot see the DLQ's operational owner/consumer, matching the ground-truth boundary.",
"Consumer processing time, DLQ monitoring, and producer behaviour are all presented as risks to verify rather than confirmed bugs.",
"Top fix is to raise DLQ MessageRetentionPeriod above the source's, recommending the 14-day maximum (1209600 s), matching the ground-truth recommendation exactly."
],
"agent_output": "# SQS Queue Audit \u2014 `order-events`\n\n---\n\n## Step 0: Parse the attributes\n\n**Source queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| `VisibilityTimeout` | `\"60\"` | 60 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\\\",\\\"maxReceiveCount\\\":5}\"` | DLQ ARN: `order-events-dlq`; `maxReceiveCount`: 5 |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**DLQ**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n| `MessageRetentionPeriod` | `\"86400\"` | 86,400 s (1 day) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n\n---\n\n## Step 1: Apply each rule\n\n### R1 \u2014 No DLQ on a processing queue\n`RedrivePolicy` is present. DLQ is wired. **No finding.**\n\n### R2 \u2014 `maxReceiveCount` outside 3\u201310\n`maxReceiveCount` = 5. Within band. **No finding.**\n\n### R3 \u2014 DLQ retention not longer than source retention\n- Source retention: **345,600 s (4 days)**\n- DLQ retention: **86,400 s (1 day)**\n- Condition to fire: DLQ retention \u2264 source retention \u2192 86,400 \u2264 345,600 \u2192 **true**\n\n**Finding: R3 (critical)**\n\n### R4 \u2014 Poison messages age out before reaching the DLQ\nArithmetic: `maxReceiveCount \u00d7 VisibilityTimeout` vs `MessageRetentionPeriod`\n\n```\n5 \u00d7 60 = 300 s\n300 s vs 345,600 s\n300 < 345,600 \u2192 inequality is NOT satisfied\n```\n\nThe product (300 s) is well within the source retention window (4 days). A poison message can exhaust all five receive attempts and still reach the DLQ with 4 days minus 5 minutes of runway remaining. **No finding.**\n\n### R5 \u2014 Visibility timeout at the 30 s default\n`VisibilityTimeout` = 60 s (not the 30 s default). **No finding.**\n\n### R6 \u2014 Retention shorter than a plausible outage\nSource `MessageRetentionPeriod` = 345,600 s (4 days). Well above the 1-hour threshold. **No finding.**\n\n### R7 \u2014 Resource policy allows a wildcard principal with no condition\n`Policy` is absent. **No finding.**\n\n### R8 \u2014 Server-side encryption at rest disabled\n`SqsManagedSseEnabled` = `\"true\"` on both queues. **No finding.**\n\n### R9 \u2014 FIFO with content-based dedup off\n`FifoQueue` attribute absent; this is a standard queue. **No finding.**\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2014 `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n\n---\n\n### Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R3: DLQ retention is shorter than source retention\n\n**Attributes**\n- Source `MessageRetentionPeriod`: `345600` s **(4 days)**\n- DLQ `MessageRetentionPeriod`: `86400` s **(1 day)**\n\n**What breaks**\n\nSQS does not reset a message's `SentTimestamp` when it is moved to the DLQ. A message's age is measured from when it was first sent to the source queue, and its remaining lifetime in the DLQ is `DLQ_retention \u2212 message_age_on_arrival`.\n\nConsider a message that fails its fifth (final) receive attempt near the end of its window in the source queue \u2014 for example, after 3 days. When it lands in the DLQ it is already 3 days old. With a DLQ retention of only 1 day, that message is deleted within hours of arrival. The DLQ appears correctly wired; the ApproximateNumberOfMessages counter may even show messages present. The ones you most need to inspect \u2014 late, repeated failures \u2014 are the ones that expire silently on arrival.\n\nIn the current configuration the gap is severe: a message could arrive in the DLQ with up to **3 days of age already accumulated** (source retention 4 days, minus the minimum time needed to exhaust 5 receive attempts at 60 s each, which is negligible), and be deleted the same day.\n\n**Recommendation**\n\nSet the DLQ's `MessageRetentionPeriod` to **strictly greater than** the source queue's retention. The safe rule of thumb is DLQ retention = source retention + a comfortable investigation buffer. A minimum safe value here is anything above 345,600 s; a practical choice is **1,209,600 s (14 days)**, which is the SQS maximum and is the standard recommendation for a DLQ that must reliably hold post-mortem evidence.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n --attributes MessageRetentionPeriod=1209600\n```\n\n---\n\n### No further findings\n\nAll other rules pass:\n\n| Rule | Result | Notes |\n|---|---|---|\n| R1 | \u2705 Pass | DLQ is wired |\n| R2 | \u2705 Pass | `maxReceiveCount` = 5, within 3\u201310 band |\n| R4 | \u2705 Pass | 5 \u00d7 60 s = 300 s \u226a 345,600 s retention; poison messages reach the DLQ with ample time remaining |\n| R5 | \u2705 Pass | Visibility timeout is 60 s, not the 30 s default |\n| R6 | \u2705 Pass | Source retention = 4 days, above the 1-hour minimum |\n| R7 | \u2705 Pass | No resource policy present |\n| R8 | \u2705 Pass | SQS-managed SSE enabled on both queues |\n| R9 | \u2705 Pass | Standard queue, FIFO dedup contract is not applicable |\n\n---\n\n### Boundary\n\nThe following questions are live risks for this queue that **this audit cannot answer**, because the answers require joins across resources, time-series data, or the systems on either side of the queue.\n\n1. **Consumer processing time vs visibility timeout.** The 60 s visibility timeout passes the \"not at default\" check, but whether it is long enough depends entirely on how long the order-event worker takes. If processing takes longer than 60 s under load, the message reappears and is delivered twice \u2014 a duplicate order event. This requires a CloudWatch `ApproximateAgeOfOldestMessage` trace or a timed sample of consumer execution, neither of which is a queue attribute.\n\n2. **Whether anyone is monitoring or draining the DLQ.** There are currently **37 messages** in `order-events-dlq`. The queue has no consumer configured in this attribute set, and there is no CloudWatch alarm configuration visible here. R3 above makes this urgent: messages in the DLQ right now are accumulating age and will be silently deleted within 1 day of their original send time. Whether an alert fires, and whether a human or automated process is draining and inspecting the DLQ, is outside this audit.\n\n3. **Effective access control (the IAM union).** There is no resource policy on the source queue, so access is governed entirely by identity-based IAM policies on the principals that produce and consume from it. This audit cannot read those policies. The effective set of principals that can send to or receive from `order-events` is invisible here. A resource policy is not mandatory, but the absence of one means there is no queue-side guard; access depends entirely on the IAM posture of every role in the account.\n\n4. **Producer behaviour \u2014 are the right services writing, with the right attributes?** Whether order events are being published by the expected services, at the expected volume, and with valid message bodies and attributes (e.g., correct `MessageGroupId` patterns if this queue is ever converted to FIFO) is a property of the producers, not the queue.\n\n5. **Live behaviour \u2014 is processing keeping up?** There are **88 messages** in flight on the source queue. Whether this represents normal depth or a growing backlog requires a time-series from CloudWatch (`ApproximateNumberOfMessages`, `NumberOfMessagesSent`, `NumberOfMessagesDeleted`). A growing backlog with a 4-day retention window is survivable; the same backlog with the current 1-day DLQ retention (R3) means the oldest failing messages are already at risk.",
"elapsed_s": 55.2
},
{
"queue": "04-poison-ages-out-before-dlq",
"condition": "control",
"trial": 0,
"score": 5,
"scores_by_item": [
1,
1,
1,
1,
0,
0,
1
],
"notes": [
"Agent correctly parses maxReceiveCount (1000), VisibilityTimeout (900s), and MessageRetentionPeriod (345600s) from the config.",
"Agent identifies the core defect: 1000 \u00d7 900s = 900000s far exceeds 345600s retention, so poison messages are deleted before reaching the DLQ \u2014 matching R2 and R4 in substance.",
"No false positives introduced; the additional findings (DLQ visibility timeout, short polling, KMS, FIFO) are soft flags, not assertions of misconfiguration absent from ground truth.",
"Agent headlines the silent message-loss defect as CRITICAL and leads with it, correctly prioritizing it above all other findings.",
"Agent does not name any specific thing it cannot determine from config alone (e.g., actual receive cadence or consumer behavior) that matches the ground-truth boundary statement.",
"Several flags (DLQ visibility timeout, FIFO/idempotency need) are presented as confirmed issues rather than 'verify with consumer/operational context' flags, violating honesty on soft flags.",
"Top fix correctly recommends lowering maxReceiveCount to 3\u20135 so messages reach the DLQ well within the retention window, matching the ground-truth recommendation in substance."
],
"agent_output": "## SQS Audit: `ledger-reconcile`\n\n---\n\n### CRITICAL \u2014 `maxReceiveCount: 1000`\n\n**What it is:** The redrive policy only moves a message to the DLQ after it has been received and failed **1,000 times**.\n\n**Why it matters:** For a daily reconciliation worker, a poison-pill message (malformed payload, unresolvable reference, etc.) will be retried up to 1,000 times before being quarantined. With a `VisibilityTimeout` of 900 seconds, a single stuck message can consume worker cycles for up to **10,000,000 seconds (~115 days)** before reaching the DLQ \u2014 far longer than the 4-day `MessageRetentionPeriod`, meaning **it will be silently dropped from the source queue before it ever reaches the DLQ**. This is a message loss guarantee, not a safety net.\n\n**Fix:**\n```\nmaxReceiveCount: 3 to 5\n```\nFor a once-daily job, even `3` is generous. Set it low enough that a bad message reaches the DLQ well within the source queue's retention window.\n\n---\n\n### HIGH \u2014 Source queue `MessageRetentionPeriod: 345600` (4 days) is shorter than the DLQ retention (14 days) but interacts lethally with `maxReceiveCount: 1000`\n\n**What it is:** This is a consequence of the above, but worth calling out independently. The source queue retains messages for only **4 days**. With `maxReceiveCount: 1000` and a 900-second visibility timeout, any message that fails consistently will expire in the source queue before exhausting its receive count. SQS then **deletes it without moving it to the DLQ**. The DLQ's 14-day retention is irrelevant if messages never arrive there.\n\n**Fix:** Fix `maxReceiveCount` first (see above). Independently, consider whether 4-day retention on the source is appropriate for a daily job \u2014 if the worker is down for a long weekend, you have margin, but it is tight. 7 days (`604800`) is a safer baseline for a financial reconciliation workload.\n\n---\n\n### HIGH \u2014 `VisibilityTimeout: 900` on the DLQ is operationally misleading\n\n**What it is:** The DLQ has a 900-second (15-minute) visibility timeout. The DLQ for a reconciliation queue should typically be **human-inspected or processed by a separate alerting/replay consumer**, not a fast-polling worker. A 900-second timeout means that if an operator or automated replay consumer reads a message and crashes, it is invisible for 15 minutes before reappearing.\n\nMore importantly, if a replay consumer re-drives messages back to the source queue, the 900-second timeout on the DLQ could cause **duplicate re-drives** if the visibility timeout expires before the consumer confirms deletion.\n\n**Fix:** Align the DLQ visibility timeout with the actual consumption pattern. If it is human/tooling-driven replay, a shorter timeout (e.g., `30`\u2013`60` seconds) is more appropriate so messages surface quickly after a failed inspection attempt. If it is automated, match it to the replay consumer's processing SLA.\n\n---\n\n### MEDIUM \u2014 `ReceiveMessageWaitTimeSeconds: 0` on the DLQ (short polling)\n\n**What it is:** The DLQ uses short polling. This means consumers checking the DLQ will frequently get empty responses even when messages exist (SQS short polling samples only a subset of servers), leading to **false negatives, missed alerts, and wasted API calls**.\n\n**Why it matters for a DLQ specifically:** DLQ monitoring is often the only signal that reconciliation is silently failing. A short-polling consumer or CloudWatch alarm driven by `ApproximateNumberOfMessages` may lag or miss spikes.\n\n**Fix:**\n```\nReceiveMessageWaitTimeSeconds: 20\n```\nEnable long polling on the DLQ, consistent with the source queue. Also ensure a CloudWatch alarm exists on `ApproximateNumberOfMessagesVisible >= 1` for the DLQ with a financial-appropriate notification threshold.\n\n---\n\n### MEDIUM \u2014 Encryption is `SqsManagedSseEnabled: true` (SSE-SQS), not SSE-KMS\n\n**What it is:** Both queues use SQS-managed encryption keys rather than a customer-managed KMS key (CMK).\n\n**Why it matters for a ledger/financial queue:** With SSE-SQS you have no key rotation control, no CloudTrail visibility into key usage, no ability to enforce key access policies, and no ability to revoke access by disabling the key. For a queue processing financial reconciliation data this is likely a compliance gap (PCI-DSS, SOC 2, internal data classification policies).\n\n**Fix:** Migrate to SSE-KMS with a CMK:\n```json\n\"KmsMasterKeyId\": \"arn:aws:kms:eu-west-1:211125758836:key/<your-cmk-id>\"\n```\nSet a key rotation policy and scope the KMS key policy to only the reconciliation worker's IAM role and the DLQ's role.\n\n---\n\n### LOW \u2014 No `MessageGroupId` / no FIFO queue for a financial reconciliation workload\n\n**What it is:** This is a standard queue. Standard queues offer **at-least-once delivery and best-effort ordering**. For a ledger reconciliation worker, duplicate message delivery means the same reconciliation operation could be applied twice.\n\n**Why it may matter:** Depends entirely on whether the consumer is idempotent. If the reconciliation worker is not strictly idempotent (most ledger operations are not by default), duplicate delivery from a standard queue is a correctness risk, not just an operational nuisance.\n\n**Fix:** Evaluate whether this should be a FIFO queue (`ledger-reconcile.fifo`). FIFO gives exactly-once processing within a 5-minute deduplication window and strict ordering. If a FIFO migration is too disruptive, ensure the consumer is rigorously idempotent with deduplication keyed on a stable message attribute.\n\n---\n\n### Summary Table\n\n| # | Severity | Issue | Fix |\n|---|----------|-------|-----|\n| 1 | **CRITICAL** | `maxReceiveCount: 1000` \u2014 poison messages expire before reaching DLQ, silent loss | Set to `3`\u2013`5` |\n| 2 | **HIGH** | Source retention (4d) + high receive count = messages deleted before DLQ redirect | Fix receive count; consider raising retention to 7d |\n| 3 | **HIGH** | DLQ `VisibilityTimeout: 900` causes duplicates on replay consumer crash | Lower to `30`\u2013`60`s or match replay SLA |\n| 4 | **MEDIUM** | DLQ uses short polling \u2014 missed alerts on failures | Set `ReceiveMessageWaitTimeSeconds: 20` |\n| 5 | **MEDIUM** | SSE-SQS not SSE-KMS \u2014 no audit trail or key control for financial data | Migrate to CMK-backed SSE-KMS |\n| 6 | **LOW** | Standard queue \u2014 at-least-once delivery risks duplicates on financial operations | Evaluate FIFO or enforce consumer idempotency |\n\n---\n\n**The most urgent fix is #1.** In its current state, the DLQ provides no protection whatsoever for a daily worker \u2014 it is effectively disabled by the combination of `maxReceiveCount: 1000` and a 4-day retention period.",
"elapsed_s": 46.8
},
{
"queue": "04-poison-ages-out-before-dlq",
"condition": "control",
"trial": 1,
"score": 5,
"scores_by_item": [
1,
1,
1,
1,
0,
0,
1
],
"notes": [
"Correctly parses RedrivePolicy maxReceiveCount=1000 and VisibilityTimeout=900s, and MessageRetentionPeriod=345600s.",
"Identifies the core defect: 1000\u00d7900s=900000s exceeds retention 345600s, so poison messages expire before reaching the DLQ \u2014 matching R2 and R4 in substance.",
"No false positives such as calling a scoped policy 'public' or inventing misconfigured resources not present in ground truth.",
"Ranks the silent-message-loss defect (maxReceiveCount\u00d7VisibilityTimeout > retention) as CRITICAL and leads with it.",
"Does not explicitly name any boundary it cannot cross from config alone (e.g., does not mention inability to observe real receive cadence or consumer behavior as an unknowable).",
"Several findings (DLQ short polling, missing alarms, SSE-SQS) are presented as confirmed bugs rather than flags to verify given consumer/operational context.",
"Top fix correctly recommends lowering maxReceiveCount (to 3\u20135) so the product stays well under retention, matching the ground-truth fix in substance."
],
"agent_output": "## SQS Audit: `ledger-reconcile`\n\n---\n\n### CRITICAL: `maxReceiveCount` is 1000\n\n**Config:** `RedrivePolicy.maxReceiveCount = 1000`\n\n**Problem:** A message must fail processing **1000 times** before it is moved to the DLQ. With a `VisibilityTimeout` of 900 seconds, a single stuck message occupies a worker slot for up to **10.4 days** of cumulative processing attempts before it is ever quarantined. In practice, poison-pill messages will be retried thousands of times, burning worker capacity and masking the real failure. For a daily reconciliation job this almost certainly means a broken message recirculates for longer than the entire `MessageRetentionPeriod` (4 days), so it will be **silently dropped by retention expiry before it ever reaches the DLQ.**\n\n**The silent-drop math:**\n- Retention = 345,600 s (4 days)\n- Each failed attempt holds the message for 900 s of visibility timeout, plus re-queue time\n- 1000 \u00d7 900 s = 900,000 s > 345,600 s \u2192 message expires in the source queue before hitting the DLQ threshold\n\n**Fix:** Set `maxReceiveCount` to a small number reflecting real retry intent. For a reconciliation worker where failures are meaningful:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}'\n```\n\nA value of **3\u20135** is appropriate for most reconciliation workloads. Tune upward only with explicit justification.\n\n---\n\n### HIGH: Source queue `MessageRetentionPeriod` is only 4 days\n\n**Config:** `MessageRetentionPeriod = 345600` (4 days)\n\n**Problem:** This is the SQS **default**, and it is short for a ledger reconciliation workload. If the daily worker is down, degraded, or behind for more than 4 days (incident, holiday freeze, deployment rollback), messages are **permanently and silently dropped** with no DLQ safety net (because retention expiry bypasses the redrive mechanism entirely \u2014 SQS does not move expired messages to the DLQ).\n\n**Fix:** Increase to the SQS maximum of 14 days to maximise recovery windows. The DLQ is already correctly set to 14 days \u2014 the source queue should match or be close to it:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attributes '{\"MessageRetentionPeriod\":\"1209600\"}'\n```\n\n---\n\n### HIGH: DLQ `ReceiveMessageWaitTimeSeconds` is 0 (short polling)\n\n**Config:** DLQ `ReceiveMessageWaitTimeSeconds = 0`\n\n**Problem:** The DLQ is configured for **short polling**. Any consumer or alarm-checking process that reads from the DLQ will receive **empty responses the majority of the time** when the queue is sparse (which a DLQ should usually be). This leads to:\n- Increased API costs from empty `ReceiveMessage` calls\n- Monitoring scripts or manual triage tooling falsely concluding the DLQ is empty when it is not (short polling samples only a subset of SQS servers per request)\n- Missed alerts on DLQ depth if the check itself uses short polling\n\nNote the source queue correctly uses long polling (`ReceiveMessageWaitTimeSeconds = 20`).\n\n**Fix:** Enable long polling on the DLQ to match the source queue:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile-dlq \\\n --attributes '{\"ReceiveMessageWaitTimeSeconds\":\"20\"}'\n```\n\n---\n\n### MEDIUM: No CloudWatch alarm on DLQ depth\n\n**Config:** Not visible in queue attributes \u2014 confirmed absent by omission.\n\n**Problem:** The DLQ currently has 0 messages, but there is no observable alerting wired to it. When messages eventually land there (and with `maxReceiveCount=1000` they may never arrive \u2014 see CRITICAL above), there is no operational signal. For a **ledger** workload, silent DLQ accumulation is a financial data integrity risk.\n\n**Fix:** Create a CloudWatch alarm on `ApproximateNumberOfMessagesVisible` for the DLQ:\n\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"ledger-reconcile-dlq-depth\" \\\n --namespace AWS/SQS \\\n --metric-name ApproximateNumberOfMessagesVisible \\\n --dimensions Name=QueueName,Value=ledger-reconcile-dlq \\\n --statistic Maximum \\\n --period 300 \\\n --threshold 1 \\\n --comparison-operator GreaterThanOrEqualToThreshold \\\n --evaluation-periods 1 \\\n --alarm-actions <your-sns-arn> \\\n --treat-missing-data notBreaching\n```\n\nThreshold of **1** is intentional \u2014 any DLQ message in a ledger system warrants immediate attention.\n\n---\n\n### MEDIUM: No alarm on source queue age (`ApproximateAgeOfOldestMessage`)\n\n**Config:** Not present.\n\n**Problem:** There are currently **26 messages** in the source queue. For a daily reconciliation worker, messages older than ~24\u201348 hours indicate the worker has stalled. Without an age alarm, this is invisible until someone manually checks. `ApproximateNumberOfMessages` alone is insufficient because it does not tell you whether messages are being processed \u2014 only that they exist.\n\n**Fix:**\n\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"ledger-reconcile-message-age\" \\\n --namespace AWS/SQS \\\n --metric-name ApproximateAgeOfOldestMessage \\\n --dimensions Name=QueueName,Value=ledger-reconcile \\\n --statistic Maximum \\\n --period 300 \\\n --threshold 172800 \\\n --comparison-operator GreaterThanOrEqualToThreshold \\\n --evaluation-periods 1 \\\n --alarm-actions <your-sns-arn>\n```\n\nThreshold of 172800 = 48 hours. Adjust to match your SLA.\n\n---\n\n### LOW: Encryption is `SqsManagedSseEnabled` (SSE-SQS), not SSE-KMS\n\n**Config:** `SqsManagedSseEnabled = true` on both queues. No `KmsMasterKeyId` present.\n\n**Problem:** SSE-SQS encrypts at rest but **does not provide**:\n- Key rotation control\n- CloudTrail audit logs of key usage (decrypt operations are invisible)\n- IAM/KMS policy-based access controls on the encryption key itself\n- Cross-account key sharing control\n\nFor a queue handling **ledger/financial data**, the inability to audit who decrypted what, and the lack of key revocation capability, is a meaningful compliance gap (PCI-DSS, SOC2, internal audit requirements).\n\n**Fix:** Migrate to SSE-KMS using a CMK. Note this requires updating any IAM roles that produce/consume messages to have `kms:GenerateDataKey` and `kms:Decrypt` permissions:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attributes '{\"KmsMasterKeyId\":\"arn:aws:kms:eu-west-1:211125758836:key/<your-cmk-id>\"}'\n```\n\nApply to the DLQ as well. Use the same CMK or a separate one depending on your key management policy.\n\n---\n\n### Summary Table\n\n| # | Finding | Severity | Impact |\n|---|---------|----------|--------|\n| 1 | `maxReceiveCount=1000` causes messages to expire before reaching DLQ | **CRITICAL** | Silent message loss |\n| 2 | Source queue retention 4 days \u2014 too short, no DLQ fallback on expiry | **HIGH** | Silent message loss on outage |\n| 3 | DLQ uses short polling | **HIGH** | Missed DLQ signals, false-empty reads |\n| 4 | No alarm on DLQ depth | **MEDIUM** | Blind to processing failures |\n| 5 | No alarm on message age | **MEDIUM** | Stalled worker goes undetected |\n| 6 | SSE-SQS instead of SSE-KMS | **LOW** | No decrypt audit trail, no key control |\n\n**Fix items 1 and 2 immediately \u2014 together they create a condition where a broken message is guaranteed to be silently dropped rather than quarantined.**",
"elapsed_s": 53.9
},
{
"queue": "04-poison-ages-out-before-dlq",
"condition": "control",
"trial": 2,
"score": 6,
"scores_by_item": [
1,
1,
1,
1,
1,
0,
1
],
"notes": [
"Correctly parses maxReceiveCount (1000) and VisibilityTimeout (900s) from the embedded RedrivePolicy and computes 900,000s vs 345,600s retention.",
"Identifies the ground-truth defect: maxReceiveCount \u00d7 VisibilityTimeout exceeds MessageRetentionPeriod, causing poison messages to expire before reaching the DLQ (R2/R4 by substance).",
"No false positives asserting misconfigurations not in ground truth; the additional flags (retention, encryption, polling) are plausible observations, not fabricated critical issues.",
"Ranks the maxReceiveCount/silent-message-loss defect as CRITICAL and headlines it first, correctly prioritizing it above all other findings.",
"Notes it cannot determine actual consumer receive cadence/job duration from config alone (VisibilityTimeout vs actual p99 job duration, CloudWatch metrics needed), matching the ground-truth boundary.",
"The '26 unprocessed messages' finding is presented as a confirmed 'live red flag' and 'anomalous' rather than clearly flagged as something that cannot be confirmed from config alone without consumer/timing context.",
"Top fix correctly recommends lowering maxReceiveCount (suggests 3\u20135) so that maxReceiveCount \u00d7 VisibilityTimeout stays well under the retention period, matching the ground-truth fix in substance."
],
"agent_output": "## SQS Audit: `ledger-reconcile`\n\n---\n\n### CRITICAL \u2014 `maxReceiveCount: 1000` is effectively disabled DLQ protection\n\n**What it is:**\nThe redrive policy sets `maxReceiveCount` to 1000, meaning a message must fail processing **1000 times** before being moved to the DLQ.\n\n**Why it matters for this queue:**\nWith a `VisibilityTimeout` of 900 seconds (15 minutes), a single message cycling through 1000 receive attempts would take:\n\n```\n1000 \u00d7 900s = 900,000 seconds \u2248 10.4 days\n```\n\nThe source queue `MessageRetentionPeriod` is only **4 days (345,600 seconds)**. A poison-pill message will be **silently dropped** before it ever reaches the DLQ. You will lose it with no record, no alert, no replay opportunity.\n\n**Fix:**\nFor a daily reconciliation worker, a value of 3\u20135 is appropriate. Set it low enough that a genuinely broken message reaches the DLQ well within the retention window:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}'\n```\n\n---\n\n### HIGH \u2014 Source queue retention period (4 days) is shorter than DLQ retention period (14 days), but not by design\n\n**What it is:**\nThe source queue retains messages for 4 days. The DLQ retains for 14 days. The asymmetry is correct in direction \u2014 the DLQ *should* be longer \u2014 but given the broken `maxReceiveCount` above, messages **never reach the DLQ** before expiring. Once that is fixed, the retention pair becomes coherent.\n\nHowever, 4 days is **low for a financial reconciliation queue**. If the consumer is down over a long weekend or incident, messages will expire silently before being processed.\n\n**Fix:**\nExtend the source queue retention to at least 7 days, ideally matching the DLQ at 14 days:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attributes '{\"MessageRetentionPeriod\":\"1209600\"}'\n```\n\n---\n\n### HIGH \u2014 26 messages sitting in the source queue with no DLQ activity\n\n**What it is:**\n`ApproximateNumberOfMessages: 26` on the source queue and `0` on the DLQ. For a *daily* reconciliation worker, 26 unprocessed messages is anomalous \u2014 it suggests either the consumer is not running, is failing silently, or messages are stuck in flight (not visible).\n\n**Why it matters:**\nThis is not a configuration value to change, but it is a live red flag in the audit. Combined with the broken `maxReceiveCount`, those 26 messages could expire before anyone notices.\n\n**Action:**\nImmediately check:\n```bash\n# Check in-flight (invisble) messages\naws sqs get-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attribute-names ApproximateNumberOfMessagesNotVisible\n\n# Check consumer is running and healthy\n# Review CloudWatch: NumberOfMessagesSent vs NumberOfMessagesDeleted over last 24h\n```\n\n---\n\n### MEDIUM \u2014 DLQ has `ReceiveMessageWaitTimeSeconds: 0` (short polling)\n\n**What it is:**\nThe DLQ is configured for **short polling**. The source queue correctly uses long polling (`ReceiveMessageWaitTimeSeconds: 20`). This inconsistency matters if any automated process or alerting lambda reads from the DLQ \u2014 it will burn API call quota and may miss messages in a distributed poll.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile-dlq \\\n --attributes '{\"ReceiveMessageWaitTimeSeconds\":\"20\"}'\n```\n\n---\n\n### MEDIUM \u2014 Encryption is SQS-managed (SSE-SQS), not KMS\n\n**What it is:**\nBoth queues use `SqsManagedSseEnabled: true`. This provides encryption at rest but with a shared AWS-managed key. For a **financial ledger reconciliation** queue, this means:\n\n- No key rotation control\n- No ability to audit key usage via CloudTrail\n- No ability to restrict decryption via IAM/KMS policy conditions\n- No envelope encryption with your own CMK\n\n**Fix:**\nMigrate to a customer-managed KMS key:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile \\\n --attributes '{\"KmsMasterKeyId\":\"arn:aws:kms:eu-west-1:211125758836:key/<your-cmk-id>\"}'\n```\n\nApply the same to the DLQ. Ensure the KMS key policy grants access to the producer and consumer IAM roles explicitly.\n\n---\n\n### LOW \u2014 No `VisibilityTimeout` review against actual job duration\n\n**What it is:**\n`VisibilityTimeout` is 900 seconds (15 minutes). This is acceptable only if every reconciliation job is **guaranteed to complete in under 15 minutes**. If the job can run longer \u2014 which is plausible for a ledger reconciliation over a large dataset \u2014 the message will become visible again mid-processing, causing a **duplicate execution**.\n\n**Fix:**\nAudit actual p99 job duration from CloudWatch or application metrics. If jobs can exceed 15 minutes, either:\n\n1. Increase the `VisibilityTimeout` to match p99 duration plus headroom, or\n2. Have the consumer call `ChangeMessageVisibility` to heartbeat and extend the timeout dynamically during processing\n\n---\n\n### Summary Table\n\n| # | Severity | Issue | Risk |\n|---|----------|-------|------|\n| 1 | **CRITICAL** | `maxReceiveCount: 1000` causes messages to expire before reaching DLQ | Silent message loss |\n| 2 | **HIGH** | Source queue retention 4 days too short for financial data | Expiry before processing |\n| 3 | **HIGH** | 26 unprocessed messages in source queue, 0 in DLQ | Active data loss in progress |\n| 4 | **MEDIUM** | DLQ uses short polling | Missed messages, API waste |\n| 5 | **MEDIUM** | SSE-SQS not SSE-KMS for financial queue | Insufficient key control/audit |\n| 6 | **LOW** | VisibilityTimeout may be shorter than job duration | Duplicate processing |\n\n**The single most urgent fix is #1.** It renders the DLQ completely non-functional and will silently destroy any poison-pill messages in a financial system.",
"elapsed_s": 42.8
},
{
"queue": "04-poison-ages-out-before-dlq",
"condition": "treatment",
"trial": 0,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON string, extracts maxReceiveCount=1000, and converts all string-typed second values to numeric form.",
"Identifies both R2 and R4 by substance, with correct arithmetic showing 900000s > 345600s causes silent message loss before DLQ.",
"No false positives; the absent resource policy is correctly noted as non-finding, and the scoped redriveAllowPolicy is not flagged as an issue.",
"R4 (silent message loss) is correctly ranked as CRITICAL while R2 is LOW, with the headline finding being the poison-message age-out.",
"Explicitly names consumer processing time vs. visibility timeout, live failure rates, and DLQ drainage as outside the config-only audit boundary.",
"VisibilityTimeout and consumer cadence questions are framed as open questions requiring metrics/consumer data, not confirmed bugs.",
"Top recommended fix is to reduce maxReceiveCount to the 3-10 band, which matches the ground-truth fix of lowering maxReceiveCount so the product stays under retention."
],
"agent_output": "# SQS Queue Audit: `ledger-reconcile`\n\n---\n\n## Step 1 \u2014 Parse the attributes\n\n**Source queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile` |\n| `VisibilityTimeout` | `\"900\"` | 900 s (15 min) |\n| `MessageRetentionPeriod` | `\"345600\"` | 345 600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `RedrivePolicy` | JSON string | `deadLetterTargetArn = \u2026:ledger-reconcile-dlq`, `maxReceiveCount = 1000` |\n| `Policy` | absent | no resource policy |\n| `FifoQueue` | absent | standard queue |\n\n**DLQ (`ledger-reconcile-dlq`)**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `MessageRetentionPeriod` | `\"1209600\"` | 1 209 600 s (14 days) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `RedriveAllowPolicy` | JSON string | `byQueue`, restricted to source ARN |\n\n---\n\n## Step 2 \u2014 Redrive path checks\n\n**R1 \u2014 No DLQ:** `RedrivePolicy` is present. \u2713 Not triggered.\n\n**R2 \u2014 `maxReceiveCount` out of band:** `maxReceiveCount = 1000`. The sane band is 3\u201310. 1 000 is far above the upper bound. \u2192 **R2 (too high) \u2014 low severity.**\n\n**R3 \u2014 DLQ retention not longer than source:**\n- Source retention: 345 600 s (4 days)\n- DLQ retention: 1 209 600 s (14 days)\n- 1 209 600 > 345 600 \u2713 Not triggered.\n\n---\n\n## Step 3 \u2014 Message lifecycle checks\n\n**R4 \u2014 Poison messages age out before the DLQ:**\n\nThe check is the inequality:\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout > MessageRetentionPeriod\n 1000 \u00d7 900 s > 345 600 s\n 900 000 s > 345 600 s\n```\n\n**900 000 s (10.4 days) > 345 600 s (4 days).** The inequality holds. \u2192 **R4 \u2014 critical.**\n\nA poison message must be received 1 000 times before it dead-letters. Each receive holds it invisible for 900 s. The minimum wall-clock to exhaust the receive count\u2014assuming a consumer picks it up again immediately after each visibility timeout\u2014is 900 000 s, which is 2.6\u00d7 the 4-day retention period. SQS will delete the message by age at the 4-day mark. It will never reach the DLQ. The DLQ will stay empty for exactly the wrong reason.\n\n**R5 \u2014 Visibility timeout at the 30 s default:** `VisibilityTimeout = 900 s`. Not at the default. \u2713 Not triggered.\n\n**R6 \u2014 Retention shorter than a plausible outage:** `MessageRetentionPeriod = 345 600 s` (4 days). Well above 1 hour. \u2713 Not triggered.\n\n---\n\n## Step 4 \u2014 Exposure checks\n\n**R7 \u2014 Open resource policy:** No `Policy` attribute present. \u2713 Not triggered.\n\n**R8 \u2014 Encryption at rest disabled:** `SqsManagedSseEnabled = true` on both queues. \u2713 Not triggered.\n\n---\n\n## Step 5 \u2014 FIFO invariants\n\n`FifoQueue` is absent; this is a standard queue. \u2713 R9 not applicable.\n\n---\n\n## Output\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2014 `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq` |\n\n---\n\n### Findings\n\n#### CRITICAL \u2014 R4: Poison messages age out before reaching the DLQ\n\n**Attributes:** `maxReceiveCount = 1000`, `VisibilityTimeout = 900 s`, `MessageRetentionPeriod = 345 600 s`\n\n**Arithmetic:**\n```\nmaxReceiveCount \u00d7 VisibilityTimeout = 1 000 \u00d7 900 = 900 000 s (\u224810.4 days)\nMessageRetentionPeriod = 345 600 s (4 days)\n900 000 s > 345 600 s \u2192 retention wins\n```\n\n**What breaks:** A message that consistently fails processing must be received 1 000 times before SQS moves it to the DLQ. Under continuous retry pressure, the lower bound on wall-clock time to exhaust that receive count is 900 000 s. The queue's retention period is 345 600 s. SQS enforces retention against the original `SentTimestamp`; the message is permanently deleted at the 4-day mark\u2014before it has any realistic chance of reaching the DLQ. The DLQ will appear correctly wired, correctly sized, and persistently empty. That empty DLQ will be interpreted as \"no failures,\" when the actual state is \"failures are being silently discarded.\" This is undetectable from the console without CloudWatch metrics.\n\n**Recommendation (three independent fixes, any one resolves the inequality):**\n\n1. **Reduce `maxReceiveCount` to the 3\u201310 band** (see R2 below). At `maxReceiveCount = 5` and `VisibilityTimeout = 900 s`, the product is 4 500 s\u2014well inside the 4-day retention. This is the highest-leverage fix because it also makes R2 disappear.\n2. **Extend `MessageRetentionPeriod` past the product.** At `maxReceiveCount = 1000`, you would need retention > 900 000 s (\u224810.4 days). SQS maximum is 1 209 600 s (14 days), so this is technically possible but leaves R2 open and means every stuck message consumes queue capacity for up to 10 days before quarantine.\n3. **Reduce `VisibilityTimeout`.** Only appropriate if the reconciliation worker genuinely processes each message in less than the reduced timeout. This is behind the boundary.\n\nFix 1 is strongly preferred. Fix 2 is a last resort. Fix 3 requires consumer data.\n\n---\n\n#### LOW \u2014 R2: `maxReceiveCount` far above the recommended band\n\n**Attribute:** `RedrivePolicy.maxReceiveCount = 1000`\n\n**Recommended band:** 3\u201310\n\n**What breaks:** At 1 000 retries, a poison message is retried continuously for as long as retention allows (and as shown in R4, retention runs out first). This also means detection of a bad message class is severely delayed\u2014by days, not minutes. The DLQ provides no early-warning signal. For a daily reconciliation worker, a `maxReceiveCount` of 1 000 has no plausible operational justification; the job runs once per day and a message that has failed 10 times across multiple job runs is almost certainly not going to succeed on attempt 1 000.\n\n**Recommendation:** Set `maxReceiveCount` to a value in the 3\u201310 band. For a daily reconciliation workload where each run is a single attempt cycle, a value of 3\u20135 is appropriate. This simultaneously resolves R4.\n\n---\n\n### Boundary\n\nThis audit is complete for what the static configuration of this queue can determine. The following questions are not answerable from these attributes and remain open:\n\n1. **Consumer processing time vs. visibility timeout.** The 900 s visibility timeout is not at the 30 s default and was presumably set deliberately. Whether 900 s is actually sufficient for the reconciliation worker\u2014or whether slow runs cause mid-processing redeliveries\u2014depends on the worker's execution time, which is a CloudWatch / application metric, not a queue attribute.\n\n2. **Why `maxReceiveCount` is 1 000.** This value is unusual enough that it may reflect a conscious decision (e.g., \"retry indefinitely until the accounting system comes back\"). If so, the underlying intent should be achieved differently\u2014a separate retry loop, a Step Functions wait state, or an exponential-backoff wrapper\u2014rather than by exploiting SQS retry count. The queue configuration alone cannot reveal the intent.\n\n3. **DLQ drainage.** The DLQ has `ApproximateNumberOfMessages = 0`. Given R4, an empty DLQ is not evidence that no messages are failing; it may be evidence that the R4 bug is active. Whether anyone is monitoring the DLQ (via a CloudWatch alarm on `ApproximateNumberOfMessages > 0` or `NumberOfMessagesSent` to the DLQ) and whether there is an operational process to inspect and redrive DLQ messages are both outside this audit.\n\n4. **The full access-control picture.** No resource policy is present on the source queue. The effective set of principals that can send to or receive from this queue is determined entirely by identity-based IAM policies attached to roles and users in account `211125758836`. Those are not visible here. The absence of a resource policy is not a finding, but it means this audit cannot confirm that only the intended reconciliation worker and its upstream producer have access.\n\n5. **Producer identity and send volume.** `ApproximateNumberOfMessages = 26` at the time of capture. Whether that represents a normal backlog for a daily job, messages from a previous failed run, or unexpected accumulation is not determinable from the attribute snapshot. The producer(s) and their send cadence are outside this audit.\n\n6. **Live failure rate.** Whether R4 is currently causing message loss is a CloudWatch question: `NumberOfMessagesDeleted` minus `NumberOfMessagesSent` to the DLQ over the retention window. This audit proves the configuration creates the conditions for silent loss; it cannot confirm whether loss is occurring right now.\n\n---\n\n**Summary.** One critical finding (R4) and one low finding (R2) are related and share a single root cause: `maxReceiveCount = 1000` is set far too high for this workload. Reducing it to the 3\u201310 band resolves both findings simultaneously. All other configuration attributes\u2014DLQ wiring, retention ordering, encryption, resource policy, FIFO invariants\u2014are either correct or not applicable.",
"elapsed_s": 60.8
},
{
"queue": "04-poison-ages-out-before-dlq",
"condition": "treatment",
"trial": 1,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses maxReceiveCount=1000 and VisibilityTimeout=900s from the embedded JSON RedrivePolicy and string-typed attributes.",
"Identifies both R2 and R4 by substance: maxReceiveCount\u00d7VisibilityTimeout (900,000s) exceeds MessageRetentionPeriod (345,600s), causing poison messages to age out before reaching the DLQ.",
"No false positives introduced; does not misclassify the scoped RedriveAllowPolicy or absent resource policy as misconfiguration.",
"R4 (silent message loss) is headlined as CRITICAL while R2 is correctly downgraded to LOW, matching the ground-truth severity ranking.",
"Explicitly names consumer processing time vs. VisibilityTimeout and real receive cadence as things that cannot be determined from config alone, matching the ground-truth boundary.",
"Items like consumer processing time, DLQ monitoring, and producer behavior are correctly presented as things to verify rather than confirmed bugs.",
"Top fix matches ground truth: lower maxReceiveCount (to 3\u201310 band) so that maxReceiveCount\u00d7VisibilityTimeout stays well under MessageRetentionPeriod, with optional retention increase as a secondary lever."
],
"agent_output": "# SQS Queue Audit: `ledger-reconcile`\n\n---\n\n## Step 1 \u2014 Parse the Attributes\n\n**Source queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile` | standard queue (no `.fifo` suffix) |\n| `VisibilityTimeout` | `\"900\"` | 900 s (15 min) |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `\"{\\\"deadLetterTargetArn\\\":\\\"...ledger-reconcile-dlq\\\",\\\"maxReceiveCount\\\":1000}\"` | DLQ wired; `maxReceiveCount` = 1,000 |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS enabled |\n| `Policy` | absent | no resource policy |\n| `FifoQueue` | absent | standard queue |\n\n**Dead-letter queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq` | standard queue |\n| `MessageRetentionPeriod` | `\"1209600\"` | 1,209,600 s (14 days) |\n| `VisibilityTimeout` | `\"900\"` | 900 s |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS enabled |\n| `RedriveAllowPolicy` | `byQueue`, source = `ledger-reconcile` | locked to this source; correct |\n\n---\n\n## Step 2 \u2014 Audit the Redrive Path\n\n**R1 \u2014 No DLQ:** Not applicable. `RedrivePolicy` is present; DLQ is wired.\n\n**R2 \u2014 `maxReceiveCount` outside the 3\u201310 band:**\n`maxReceiveCount` = **1,000**. This is above the upper end of the 3\u201310 band. Classify R2 as **low** (too high).\n\n**R3 \u2014 DLQ retention not longer than source retention:**\n- Source `MessageRetentionPeriod`: 345,600 s (4 days)\n- DLQ `MessageRetentionPeriod`: 1,209,600 s (14 days)\n- 1,209,600 > 345,600 \u2713\n\nR3 does **not** fire. A message that enters the DLQ at the last moment of the source window still has at least 10 days of headroom in the DLQ before its original `SentTimestamp` causes it to age out.\n\n---\n\n## Step 3 \u2014 Audit the Message Lifecycle\n\n**R4 \u2014 Poison messages age out before reaching the DLQ:**\n\nThe inequality to check: `maxReceiveCount \u00d7 VisibilityTimeout > MessageRetentionPeriod`\n\n```\n1,000 \u00d7 900 s = 900,000 s\nMessageRetentionPeriod = 345,600 s\n\n900,000 s > 345,600 s \u2192 TRUE\n```\n\n**R4 fires.** For a genuine poison message, SQS must deliver it 1,000 times and observe each visibility window expire before it considers the `maxReceiveCount` exhausted. The minimum wall-clock to do that is 900,000 s (~10.4 days). The source queue only retains messages for 345,600 s (4 days). Retention wins: the message is deleted by age at roughly the 4-day mark, after fewer than `\u230a345,600 \u00f7 900\u230b` = **384 receives**, long before the 1,000th receive that would trigger the redrive. The DLQ is correctly wired, correctly sized \u2014 and unreachable for any message that takes more than a handful of seconds per delivery attempt.\n\n**R5 \u2014 Visibility timeout at the 30 s default:**\n`VisibilityTimeout` = 900 s. Not the default. R5 does **not** fire.\n\n**R6 \u2014 Retention shorter than a plausible outage:**\n`MessageRetentionPeriod` = 345,600 s (4 days). Well above the 1-hour threshold. R6 does **not** fire.\n\n---\n\n## Step 4 \u2014 Audit Exposure\n\n**R7 \u2014 Open resource policy:**\n`Policy` is absent. No resource policy to audit. R7 does **not** fire.\n\n**R8 \u2014 Encryption at rest disabled:**\n`SqsManagedSseEnabled` = `\"true\"` on both source and DLQ. R8 does **not** fire.\n\n---\n\n## Step 5 \u2014 Audit FIFO Invariants\n\n**R9 \u2014 FIFO with content-based dedup off:**\nQueue is standard (no `FifoQueue` attribute). R9 does **not** apply.\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2014 `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq` |\n\n---\n\n### Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R4: Poison messages age out before reaching the DLQ\n\n**Attributes:** `maxReceiveCount` = 1,000 \u00b7 `VisibilityTimeout` = 900 s \u00b7 `MessageRetentionPeriod` = 345,600 s\n\n**Arithmetic:**\n```\nMinimum time to exhaust maxReceiveCount:\n 1,000 \u00d7 900 s = 900,000 s (~10.4 days)\n\nSource retention:\n 345,600 s (4 days)\n\n900,000 s > 345,600 s \u2192 retention expires first\n```\n\n**What breaks:** Any message that fails on every delivery attempt will be deleted by the `MessageRetentionPeriod` timer after approximately 4 days \u2014 after only ~384 of the 1,000 required receives. SQS never increments the receive count to 1,000, so the redrive condition is never met. The DLQ receives nothing. From the console, every indicator looks healthy: the DLQ is wired, retention is generous, the DLQ itself is empty. The empty DLQ is not a sign that nothing is failing; it is a sign that the DLQ is unreachable.\n\n**Fix:** The product `maxReceiveCount \u00d7 VisibilityTimeout` must be less than `MessageRetentionPeriod`. You have two independent levers \u2014 use both:\n\n1. **Reduce `maxReceiveCount`** to a value in the 3\u201310 band. For a daily reconciliation worker, 5 is a reasonable starting point: enough to absorb transient failures across a few retry cycles, fast enough to dead-letter genuine poison messages within hours.\n2. **Optionally extend source retention** if 4 days is already a hard constraint. At `maxReceiveCount` = 5 and `VisibilityTimeout` = 900 s, the product is 4,500 s (~75 min), which fits inside 4 days with enormous margin.\n\nDo not try to fix this by lowering `VisibilityTimeout` alone unless you have verified that the reconciliation worker actually completes within the shorter window; an undersized timeout re-delivers messages mid-processing (the R5 failure mode).\n\n---\n\n#### \ud83d\udfe1 LOW \u2014 R2: `maxReceiveCount` above the recommended band\n\n**Attribute:** `RedrivePolicy.maxReceiveCount` = 1,000\n\n**What breaks:** Setting `maxReceiveCount` to 1,000 is the proximate cause of R4 above and also an independent operational problem: even if retention were extended far enough to satisfy the R4 arithmetic, a poison message would be retried 1,000 times over 10+ days before reaching the DLQ, maximising the detection lag and the noise in consumer logs. For a daily job this also means a single bad message can monopolise a worker slot across hundreds of daily runs before it is quarantined.\n\n**Fix:** Reduce to the 3\u201310 band. 5 is recommended for this workload pattern (see R4 fix above). Once R4 is resolved, this finding is closed as a side-effect.\n\n---\n\n### Boundary\n\nThis audit read the static configuration of `ledger-reconcile` and `ledger-reconcile-dlq`. The two findings above are grounded entirely in those attributes and require no further evidence. The following questions are unanswerable from queue configuration alone and should be verified separately:\n\n1. **Consumer processing time vs. `VisibilityTimeout`.** The visibility timeout is 900 s (15 min). Whether the reconciliation worker completes a message within that window is a property of the worker, not the queue. If processing routinely exceeds 15 min, messages are re-delivered while still in-flight, producing duplicate processing without any failure signal. Verify against the worker's p99 processing latency in CloudWatch or application metrics. If the worker can exceed 15 min, it should extend the visibility timeout via `ChangeMessageVisibility` heartbeats rather than relying on a static queue setting.\n\n2. **Live DLQ behaviour.** `ApproximateNumberOfMessages` = 0 on the DLQ. Post-fix, this should be monitored. A DLQ that remains permanently empty after fixing R4 could mean clean processing, or it could mean the DLQ itself is not being reached for a different reason. Set a CloudWatch alarm on `ApproximateNumberOfMessages` > 0 on the DLQ so any future redrive events are visible.\n\n3. **Who is draining the DLQ.** The `RedriveAllowPolicy` correctly locks the DLQ to this source queue. But nothing in the queue configuration specifies whether anyone is monitoring or consuming the DLQ. A dead-letter queue that is never read is a silent graveyard. Confirm there is an operational process \u2014 alert, runbook, or automated reprocessing \u2014 that acts on DLQ arrivals.\n\n4. **IAM access beyond the resource policy.** There is no resource policy on this queue, so the entire access surface is identity-based policies in the account. This audit cannot see those. The set of principals that can send to, receive from, or delete messages on this queue is determined by IAM policies attached to roles, users, and services in account `211125758836` \u2014 none of which is visible here. A least-privilege review of those policies is outside this audit's scope.\n\n5. **Producer behaviour.** Whether the correct upstream services are writing to `ledger-reconcile`, at the expected rate, with well-formed message bodies, is not encoded in the queue. The 26 in-flight messages visible at snapshot time are a point-in-time count, not a trend. Producer-side correctness needs the producer inventory and, if message format matters, a schema contract.",
"elapsed_s": 63.1
},
{
"queue": "04-poison-ages-out-before-dlq",
"condition": "treatment",
"trial": 2,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON extracting maxReceiveCount=1000 and DLQ ARN, and converts all string-typed seconds to numeric values.",
"Identifies both R2 and R4 by substance, with the exact arithmetic: 1000 \u00d7 900 = 900,000 s > 345,600 s retention, meaning poison messages age out before reaching the DLQ.",
"No false positives introduced; the scoped RedriveAllowPolicy is not mislabeled as an open policy, and no spurious findings are asserted.",
"R4 (silent message loss) is ranked CRITICAL and headlined first; R2 is ranked LOW and correctly identified as a contributing cause rather than the primary severity.",
"Explicitly names multiple things outside config-only determination, including consumer processing time vs. visibility timeout and whether the DLQ is monitored \u2014 matching the ground-truth boundary.",
"Soft flags such as consumer processing time and DLQ monitoring are presented as things to verify, not confirmed bugs.",
"Top fix recommendation matches ground truth: lower maxReceiveCount (or raise retention) so maxReceiveCount \u00d7 VisibilityTimeout stays well under MessageRetentionPeriod, with a concrete example of maxReceiveCount=5."
],
"agent_output": "# SQS Queue Audit: `ledger-reconcile`\n\n---\n\n## Step 1 \u2014 Parse the Attributes\n\n**Source queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile` | \u2014 |\n| `VisibilityTimeout` | `\"900\"` | 900 s (15 min) |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS enabled |\n| `RedrivePolicy` | `\"{\\\"deadLetterTargetArn\\\":\\\"\u2026:ledger-reconcile-dlq\\\",\\\"maxReceiveCount\\\":1000}\"` | DLQ = `ledger-reconcile-dlq`; `maxReceiveCount` = 1,000 |\n| `Policy` | absent | no resource policy |\n| `FifoQueue` | absent | Standard queue |\n| `KmsMasterKeyId` | absent | \u2014 |\n\n**Dead-letter queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq` | \u2014 |\n| `VisibilityTimeout` | `\"900\"` | 900 s |\n| `MessageRetentionPeriod` | `\"1209600\"` | 1,209,600 s (14 days) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS enabled |\n| `RedriveAllowPolicy` | parsed | permits redrive from source ARN only |\n\n---\n\n## Step 2 \u2014 Apply the Rules\n\n### R1 \u2014 No DLQ on a processing queue\n**Not triggered.** `RedrivePolicy` is present. DLQ is `ledger-reconcile-dlq`.\n\n### R2 \u2014 `maxReceiveCount` outside the 3\u201310 band\n`maxReceiveCount` = **1,000**.\nUpper bound of the sane band is 10. 1,000 >> 10. \u2192 **R2 fires (low; too high).**\n\n### R3 \u2014 DLQ retention not longer than source retention\n- Source `MessageRetentionPeriod`: 345,600 s (4 days)\n- DLQ `MessageRetentionPeriod`: 1,209,600 s (14 days)\n- 1,209,600 > 345,600 \u2713\n\n**Not triggered.** DLQ retention is correctly longer than source retention.\n\n### R4 \u2014 Poison messages age out before reaching the DLQ\nThe inequality to evaluate:\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout > MessageRetentionPeriod\n 1,000 \u00d7 900 s > 345,600 s\n 900,000 s > 345,600 s\n```\n\n**900,000 > 345,600 \u2014 R4 fires (critical).**\n\nA poison message requires at least 900,000 seconds (\u2248 10.4 days) of wall-clock time to exhaust all 1,000 receive attempts. The source queue's retention is only 345,600 seconds (4 days). Retention wins: the message is deleted by age on day 4. The DLQ is correctly wired and sized, but it is unreachable for any message that fails consistently \u2014 precisely the class of message it exists to catch.\n\n### R5 \u2014 Visibility timeout at the 30 s default\n`VisibilityTimeout` = 900 s. **Not triggered.**\n\n### R6 \u2014 Retention shorter than a plausible outage\nSource `MessageRetentionPeriod` = 345,600 s (4 days). Well above the 1-hour threshold. **Not triggered.**\n\n### R7 \u2014 Open resource policy\nNo `Policy` attribute present. **Not triggered.**\n\n### R8 \u2014 Encryption at rest disabled\n`SqsManagedSseEnabled` = `\"true\"`. **Not triggered.**\n\n### R9 \u2014 FIFO deduplication contract\n`FifoQueue` absent (Standard queue). **Not triggered.**\n\n---\n\n## Step 3 \u2014 Rank and Report\n\n---\n\n## Output\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2192 `arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq` |\n\n---\n\n### Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R4: Poison messages age out before reaching the DLQ\n\n**Attributes involved:** `VisibilityTimeout` (900 s), `RedrivePolicy.maxReceiveCount` (1,000), `MessageRetentionPeriod` (345,600 s)\n\n**What breaks:**\n\nThe minimum wall-clock time for a consistently-failing message to exhaust its receive count is:\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout = 1,000 \u00d7 900 s = 900,000 s (\u2248 10.4 days)\n```\n\nThe source queue's `MessageRetentionPeriod` is 345,600 s (4 days). Because 900,000 s > 345,600 s, retention expires first. A poison message \u2014 a malformed ledger event, a record that triggers a downstream constraint violation, anything that fails on every attempt \u2014 is silently deleted on day 4. The DLQ receives nothing. The queue's \"generous retention\" and correctly-configured DLQ together create the appearance of a safe setup while delivering zero quarantine for the failures that matter most.\n\n**Recommendation:**\n\nEither lower `maxReceiveCount` so the product falls inside the retention window, or extend `MessageRetentionPeriod` so it exceeds the product, or both. Given the daily-reconciliation workload, a `maxReceiveCount` in the 3\u201310 band is almost certainly sufficient \u2014 a legitimate reconciliation job does not need 1,000 independent delivery attempts before quarantine. Concretely:\n\n```\nmaxReceiveCount = 5, VisibilityTimeout = 900 s\n\u2192 5 \u00d7 900 = 4,500 s << 345,600 s \u2713\n```\n\nIf the intention is to permit extended retry over days, extend `MessageRetentionPeriod` instead. Either way, verify the product satisfies `maxReceiveCount \u00d7 VisibilityTimeout < MessageRetentionPeriod` after the change.\n\n---\n\n#### \ud83d\udfe1 LOW \u2014 R2: `maxReceiveCount` far above the recommended band\n\n**Attributes involved:** `RedrivePolicy.maxReceiveCount` (1,000)\n\n**What breaks:**\n\nThe recommended band for `maxReceiveCount` is 3\u201310. At 1,000, a poison message is retried for 10+ days of continuous visibility-timeout cycles before it is quarantined. This is the direct contributor to R4 above. Even if R4 were resolved by extending retention, a `maxReceiveCount` of 1,000 delays detection of a broken message class by an operationally unreasonable interval, masks failures in alerting, and means the DLQ is effectively never populated during an incident \u2014 the opposite of its purpose.\n\n**Recommendation:**\n\nFor a daily reconciliation worker, a `maxReceiveCount` of 3\u20135 is appropriate: it tolerates transient infrastructure blips (network hiccup, downstream timeout) without dead-lettering good messages, while quarantining genuine poison messages within minutes rather than weeks. If the queue was set to 1,000 to avoid the DLQ during normal operation, the correct fix is to address whatever is causing spurious failures, not to set `maxReceiveCount` high enough that they never surface.\n\n---\n\n### Boundary\n\nThe following questions are outside what a single queue's configuration can answer. They are stated explicitly because a clean-looking config can still be part of a broken system.\n\n1. **Consumer processing time vs. visibility timeout.** The 900 s visibility timeout is not at the 30 s default, which is good, but whether 900 s is actually long enough depends on how long the reconciliation worker takes per message. If a reconciliation job runs longer than 15 minutes without calling `ChangeMessageVisibility`, the message reappears and is delivered twice. This audit cannot see consumer processing time \u2014 that is in CloudWatch (Lambda duration, ECS task duration, or application metrics), not in the queue attributes.\n\n2. **Whether the DLQ is monitored and drained.** The DLQ currently holds 0 messages. That is consistent with correct operation, but it is also consistent with R4 suppressing all DLQ deliveries (messages deleted by age before they arrive). Whether any message has ever reached the DLQ, and whether there is an alert on `ApproximateNumberOfMessages` > 0 on the DLQ, requires CloudWatch and the team's alerting config \u2014 neither is a queue attribute.\n\n3. **Live message behaviour.** The source queue has 26 messages in flight or waiting. Whether those are healthy, stalled, or repeatedly failing is a time-series question. `ApproximateNumberOfMessages` is a snapshot; the age of the oldest message (`ApproximateAgeOfOldestMessage`) and redrive metrics are in CloudWatch, not here.\n\n4. **Effective IAM permissions.** There is no resource policy on this queue, which removes one exposure vector, but the effective set of principals that can `SendMessage`, `ReceiveMessage`, or `DeleteMessage` is the union of this queue's resource policy (empty) and every identity policy in account `211125758836`. Those identity policies are not visible in `GetQueueAttributes`. A principal with an IAM policy granting `sqs:*` on `*` has full access regardless of the resource policy audit above.\n\n5. **Producer contract.** Whether the right services are writing to this queue, whether they are setting `MessageGroupId` or `MessageDeduplicationId` correctly (not applicable here \u2014 Standard queue \u2014 but relevant if this queue is ever migrated to FIFO), and whether any producer is silently swallowing send errors, requires inspection of the producers on the other side of the queue.\n\n6. **DLQ consumer and operational runbook.** The DLQ is wired and has longer retention than the source. Whether anyone is alerted when it receives a message, and whether there is a runbook for inspecting and replaying DLQ messages, is an operational question this audit cannot answer.\n\n---\n\n**Summary:** one critical finding (R4 \u2014 silent message loss by arithmetic), one low finding (R2 \u2014 `maxReceiveCount` of 1,000 is the root cause). The fix is the same for both: bring `maxReceiveCount` into the 3\u201310 band. Everything else in the configuration is correctly set.",
"elapsed_s": 63.1
},
{
"queue": "05-default-visibility-short-retention",
"condition": "control",
"trial": 0,
"score": 3,
"scores_by_item": [
1,
1,
0,
0,
0,
0,
1
],
"notes": [
"Agent correctly identifies both R5 (VisibilityTimeout 30s) and R6 (MessageRetentionPeriod 300s), implying correct parsing of string-typed seconds and RedrivePolicy.",
"Both ground-truth defects R5 and R6 are identified by substance.",
"Agent introduces multiple false positives not in ground truth: missing KMS key, missing resource-based policy, DLQ short polling, and maxReceiveCount interaction \u2014 none are ground-truth findings.",
"Agent escalates the soft R6 flag to 'CRITICAL' and treats R5 as 'HIGH confirmed bug', violating the requirement not to headline soft flags as confirmed critical defects.",
"Agent does not name any boundary it cannot determine from config alone; it presents all findings as confirmed issues without acknowledging the consumer p99 / outage-duration unknowns.",
"Agent presents both R5 and R6 as confirmed bugs rather than soft flags to verify \u2014 explicitly calling retention 'almost certainly a misconfiguration' and visibility 'causing duplicate processing' without qualifying them as dependent on consumer behavior.",
"Top fix recommendation (raise retention to cover plausible outage; set VisibilityTimeout above consumer p99) matches ground-truth substance."
],
"agent_output": "## SQS Queue Audit: `click-events`\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 `MessageRetentionPeriod` is 300 seconds (5 minutes) on source queue\n\n**What it means:**\nAny message that is not consumed within 5 minutes of being enqueued is silently deleted by SQS \u2014 regardless of whether it was ever received. This is almost certainly a misconfiguration. The default is 4 days; the minimum is 60 seconds, maximum 14 days.\n\n**The risk:**\nAt ~1,190 messages currently sitting in the queue, if your consumers lag, stall, or deploy slowly, messages are being permanently dropped with no error, no alarm, and no recovery path. They will never reach the DLQ \u2014 retention expiry is not a delivery failure, so `maxReceiveCount` does not apply. Those messages are gone.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes MessageRetentionPeriod=345600 # 4 days, or set to your SLA\n```\nSet this to at minimum several hours, realistically 4 days, to give you time to recover from a consumer outage.\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 `VisibilityTimeout` (30s) is likely too short for a high-volume analytics ingest queue\n\n**What it means:**\nOnce a consumer receives a message, other consumers are blocked from seeing it for only 30 seconds. If processing takes longer than 30 seconds \u2014 including any downstream write latency, batching, retries to a data store \u2014 SQS will make the message visible again and another consumer will receive it. This causes **duplicate processing**.\n\n**The risk:**\nFor analytics ingest, duplicates inflate counts and distort metrics. At high volume this is not an edge case. There is no indication consumers are calling `ChangeMessageVisibility` to extend the timeout, and for a high-throughput queue that pattern adds complexity and failure modes of its own.\n\n**Fix:**\nProfile your actual p99 processing time end-to-end, then set the timeout to at least 2\u20133\u00d7 that value. For most analytics pipelines writing to S3, Kinesis, or a database, 5\u201310 minutes is more appropriate:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes VisibilityTimeout=300 # adjust to your measured p99 + margin\n```\nThe DLQ `VisibilityTimeout` should match or exceed the source queue's value.\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 `maxReceiveCount: 5` may be too low given the short `VisibilityTimeout`\n\n**What it means:**\nA message is moved to the DLQ after 5 receive attempts. But because the `VisibilityTimeout` is only 30 seconds, **visibility timeout expiries count as receive attempts**. If a consumer is slow or crashes mid-processing, the message can burn through all 5 attempts without ever genuinely failing \u2014 it gets moved to the DLQ not because it is bad, but because the timeout was too tight.\n\n**The risk:**\nValid messages land in the DLQ and are never processed. Your analytics data silently has gaps. This is compounded by the 5-minute retention on the source queue \u2014 by the time you notice and attempt a redrive from the DLQ, the operational context is already lost.\n\n**Fix:**\nFix the `VisibilityTimeout` first. Once that is correct, re-evaluate `maxReceiveCount`. For analytics ingest where the message content is unlikely to be the cause of failure, a value of 3 is usually sufficient and keeps the DLQ as a genuine poison-message indicator rather than a timeout dustbin.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 No `KmsMasterKeyId`; relying on `SqsManagedSseEnabled`\n\n**What it means:**\nEncryption at rest is on, which is good. However it uses SQS-managed keys (SSE-SQS), not AWS KMS (SSE-KMS). With SSE-SQS, AWS manages the key entirely. You have no ability to audit key usage via CloudTrail, rotate the key on your own schedule, restrict access via key policy, or revoke access in an incident.\n\n**The risk:**\nFor click event data that may contain user identifiers or behavioural data, loss of key control may violate your data governance requirements or compliance posture (GDPR, SOC2, etc.). Any IAM principal that can call `sqs:ReceiveMessage` can read the data; there is no second layer of access control via the key policy.\n\n**Fix:**\nIf data classification or compliance requires it, migrate to a customer-managed KMS key:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes KmsMasterKeyId=arn:aws:kms:eu-west-1:211125758836:key/<your-key-id>\n```\nApply the same to the DLQ. Ensure your consumer and producer IAM roles have `kms:GenerateDataKey` and `kms:Decrypt` on that key.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 No Queue Access Policy (`Policy` attribute absent)\n\n**What it means:**\nThere is no resource-based policy attached to either queue. Access is controlled entirely by IAM identity policies. This is not wrong by default, but it means:\n- Any IAM principal in account `211125758836` with broad SQS permissions can read from, write to, or purge this queue\n- There is no explicit deny for cross-account access\n- There is no condition-based restriction (e.g., `aws:SourceVpc`, `aws:SourceArn`)\n\n**The risk:**\nFor a high-volume ingest queue, an overly permissioned role or a misconfigured producer could inject malformed messages or drain the queue. Without a policy, there is no audit boundary at the resource level.\n\n**Fix:**\nAttach a least-privilege resource policy. At minimum, explicitly restrict `sqs:SendMessage` to the known producer ARN and `sqs:ReceiveMessage`/`sqs:DeleteMessage` to the known consumer role:\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowProducer\",\n \"Effect\": \"Allow\",\n \"Principal\": { \"AWS\": \"arn:aws:iam::211125758836:role/click-event-producer\" },\n \"Action\": \"sqs:SendMessage\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\"\n },\n {\n \"Sid\": \"AllowConsumer\",\n \"Effect\": \"Allow\",\n \"Principal\": { \"AWS\": \"arn:aws:iam::211125758836:role/click-event-consumer\" },\n \"Action\": [\"sqs:ReceiveMessage\", \"sqs:DeleteMessage\", \"sqs:ChangeMessageVisibility\", \"sqs:GetQueueAttributes\"],\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\"\n }\n ]\n}\n```\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 DLQ has `ReceiveMessageWaitTimeSeconds: 0` (short polling)\n\n**What it means:**\nThe DLQ is configured for short polling. This is a minor cost and reliability issue rather than a data-loss risk, but it matters operationally. Any process that monitors or redrives the DLQ will make many empty API calls, increasing cost and potentially missing messages that are on a subset of SQS servers.\n\n**The risk:**\nLow, but if you have automation that checks the DLQ and triggers a redrive or alert, it may intermittently miss messages and require multiple polls to see them consistently.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events-dlq \\\n --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n### Summary Table\n\n| # | Severity | Issue | Risk |\n|---|----------|-------|------|\n| 1 | \ud83d\udd34 Critical | `MessageRetentionPeriod` = 5 minutes | Silent message drop |\n| 2 | \ud83d\udfe0 High | `VisibilityTimeout` = 30s, likely too short | Duplicate processing |\n| 3 | \ud83d\udfe0 High | `maxReceiveCount` = 5 interacts badly with short timeout | Valid messages sent to DLQ |\n| 4 | \ud83d\udfe1 Medium | SSE-SQS not SSE-KMS | No key audit/control |\n| 5 | \ud83d\udfe1 Medium | No resource-based queue policy | Overly broad access surface |\n| 6 | \ud83d\udfe1 Medium | DLQ on short polling | Missed messages in monitoring |\n\n---\n\n**Immediate action required on item 1.** The 5-minute retention period is almost certainly causing live data loss right now.",
"elapsed_s": 52.5
},
{
"queue": "05-default-visibility-short-retention",
"condition": "control",
"trial": 1,
"score": 3,
"scores_by_item": [
1,
1,
0,
0,
0,
0,
1
],
"notes": [
"Agent correctly parses the RedrivePolicy, maxReceiveCount, VisibilityTimeout (30s), and MessageRetentionPeriod (300s) as string-typed seconds.",
"Agent identifies both R5 (VisibilityTimeout at 30s default) and R6 (MessageRetentionPeriod at 300s) by substance.",
"Agent asserts a material misconfiguration (no resource-based policy = over-permissive access) that is not in the ground truth findings, constituting a false positive.",
"Agent escalates the soft flag R6 (retention) to CRITICAL and R5 to HIGH, rather than treating both as medium soft flags; ground truth says neither is a confirmed bug from config alone.",
"Agent does not name any boundary it cannot cross (e.g., consumer processing-time distribution or queue metrics over time); it asserts duplicate processing as a real risk without acknowledging it depends on consumer p99.",
"Agent presents VisibilityTimeout and retention as confirmed bugs causing active data loss, not as flags to verify; ground truth explicitly states neither is a confirmed bug from config alone.",
"Agent's top fix recommendation (raise retention to cover a plausible outage; set VisibilityTimeout deliberately above consumer p99) matches the ground-truth fix in substance."
],
"agent_output": "## SQS Queue Audit: `click-events`\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 `MessageRetentionPeriod` is 300 seconds (5 minutes) on source queue\n\n**What it means:**\nAny message not consumed within 5 minutes of being enqueued is permanently deleted by SQS, regardless of whether it was processed. For a high-volume analytics ingest queue this is almost certainly a misconfiguration, not a deliberate choice.\n\n**Why it's dangerous here specifically:**\n- `VisibilityTimeout` is 30 seconds and `maxReceiveCount` is 5, meaning a message can be retried up to 5 times with 30-second invisibility windows. That alone consumes up to ~2.5 minutes of the 5-minute retention window just in retry cycles, leaving a very thin margin.\n- Any consumer lag spike, deployment, or brief outage longer than 5 minutes causes **silent, permanent data loss** with no recovery path.\n- The DLQ has a 14-day retention period, which suggests someone configured the DLQ correctly but left the source queue at or near the SQS default minimum (60 seconds) or a test value and never corrected it.\n- `ApproximateNumberOfMessages: 1190` means there are currently ~1,190 messages sitting in the queue right now, and they are actively expiring.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes MessageRetentionPeriod=345600 # 4 days, or align with your SLA\n```\nTypical values: 345600 (4 days) or 1209600 (14 days, matching the DLQ). Choose based on how long you need a recovery window. **Do this immediately** \u2014 messages are being lost right now.\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 `VisibilityTimeout` is likely too short for retry exhaustion + DLQ path\n\n**What it means:**\n`VisibilityTimeout` is 30 seconds. With `maxReceiveCount: 5`, a message must fail 5 consecutive times before SQS routes it to the DLQ. If your consumer takes close to or more than 30 seconds to process (or fail) a message, SQS will make the message visible again before the consumer has finished, causing the same message to be received by **another consumer concurrently**.\n\n**Why it matters:**\n- This produces **duplicate processing** \u2014 a message counts as \"received\" multiple times in parallel, burning through the `maxReceiveCount` faster than intended.\n- For an analytics ingest queue, duplicate events will silently inflate your metrics unless your downstream pipeline is idempotent (and you should not assume it is).\n- 30 seconds is not unreasonable, but it must be strictly greater than your P99 consumer processing time. If consumers do any I/O (DB writes, HTTP calls to downstream), this is a real risk.\n\n**Fix:**\nSet `VisibilityTimeout` to at least 2\u20133\u00d7 your P99 consumer processing time. If consumers take up to 20 seconds, use 60\u201390 seconds. Also ensure your consumer calls `ChangeMessageVisibility` to extend the timeout if processing may exceed the configured value:\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes VisibilityTimeout=60\n```\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 No queue access policy present (implicit over-permissive default)\n\n**What it means:**\nNeither queue has a `Policy` attribute in the API response. This means **no resource-based policy is attached**. SQS falls back to IAM identity-based policies only, which means any principal in account `211125758836` with `sqs:*` or `sqs:SendMessage` / `sqs:ReceiveMessage` in their IAM policy can interact with this queue without any queue-level guard.\n\n**Why it matters:**\n- No least-privilege boundary at the queue resource level. A compromised IAM role with broad SQS permissions anywhere in the account has unrestricted access.\n- No explicit `Deny` for cross-account access or `aws:SourceArn` conditions to restrict which services can publish (e.g. if this is fed by API Gateway, Lambda, or Kinesis, those should be locked down explicitly).\n- For a queue holding PII-adjacent click-event data, this is a meaningful exposure surface.\n\n**Fix:**\nAttach a resource policy that explicitly allows only the intended producer(s) and consumer(s):\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowProducer\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": \"arn:aws:iam::211125758836:role/click-events-producer-role\"\n },\n \"Action\": \"sqs:SendMessage\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\"\n },\n {\n \"Sid\": \"AllowConsumer\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"AWS\": \"arn:aws:iam::211125758836:role/click-events-consumer-role\"\n },\n \"Action\": [\n \"sqs:ReceiveMessage\",\n \"sqs:DeleteMessage\",\n \"sqs:ChangeMessageVisibility\",\n \"sqs:GetQueueAttributes\"\n ],\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\"\n },\n {\n \"Sid\": \"DenyAllOthers\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"sqs:*\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\",\n \"Condition\": {\n \"ArnNotLike\": {\n \"aws:PrincipalArn\": [\n \"arn:aws:iam::211125758836:role/click-events-producer-role\",\n \"arn:aws:iam::211125758836:role/click-events-consumer-role\"\n ]\n }\n }\n }\n ]\n}\n```\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 `maxReceiveCount: 5` may be too high given the 5-minute retention (compounding the critical issue)\n\n**What it means:**\nWith 5 retries \u00d7 30-second visibility timeout, the theoretical maximum time a message spends in retry cycles before hitting the DLQ is **150 seconds**, on top of however long it sat in the queue before first delivery. With a 300-second retention window, a message that arrives and immediately starts failing has roughly 150 seconds of margin before it expires out of the source queue entirely \u2014 **and SQS will delete it before it ever reaches the DLQ**.\n\n**Why it matters:**\nThe DLQ is useless as a safety net if the source queue's retention period is shorter than the time needed to exhaust retries. Messages will vanish silently rather than land in the DLQ where you could inspect and replay them. This is a direct consequence of the retention misconfiguration but worth calling out explicitly because fixing retention alone doesn't mean you should leave `maxReceiveCount` unexamined.\n\n**Fix:**\nPrimarily fix retention (see Critical item). Once retention is correct, validate that `maxReceiveCount` matches your actual retry/backoff strategy. For analytics ingest, 3\u20135 is typically appropriate. If consumers are idempotent and fast, 3 is sufficient and reduces the window for duplicates.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 `ReceiveMessageWaitTimeSeconds: 0` on DLQ (minor cost/ops issue)\n\n**What it means:**\nThe DLQ has short polling enabled (0 seconds). This is the default but suboptimal. Any process or Lambda that polls the DLQ for alerting or reprocessing will issue empty API calls continuously, increasing cost and API rate consumption.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events-dlq \\\n --attributes ReceiveMessageWaitTimeSeconds=20\n```\nThe source queue already has long polling at 20 seconds, which is correct. Apply the same to the DLQ.\n\n---\n\n### \u2705 What is configured correctly\n\n| Attribute | Assessment |\n|---|---|\n| `RedrivePolicy` present | DLQ is wired up correctly |\n| `RedriveAllowPolicy` on DLQ | Correctly restricts redrive to source queue only (`byQueue`) |\n| `SqsManagedSseEnabled: true` on both queues | Encryption at rest is enabled |\n| `ReceiveMessageWaitTimeSeconds: 20` on source | Long polling correctly configured, reduces empty receives |\n| DLQ `MessageRetentionPeriod: 1209600` | 14 days is appropriate for investigation/replay |\n| Same-region, same-account DLQ | No cross-account/region DLQ complexity |\n\n---\n\n### Summary Table\n\n| Severity | Issue | Impact |\n|---|---|---|\n| \ud83d\udd34 Critical | Source queue retention is 5 minutes | Active message loss right now |\n| \ud83d\udfe0 High | VisibilityTimeout may cause concurrent redelivery | Duplicate processing |\n| \ud83d\udfe0 High | No resource-based queue policy | Over-broad access, no least-privilege boundary |\n| \ud83d\udfe1 Medium | Retry exhaustion can exceed retention window | DLQ unreachable, silent loss |\n| \ud83d\udfe1 Medium | DLQ short polling | Cost/efficiency, not a data-loss risk |\n\n**Immediate action required:** The 300-second retention period is the only issue that is actively causing data loss right now. Fix that first.",
"elapsed_s": 54.7
},
{
"queue": "05-default-visibility-short-retention",
"condition": "control",
"trial": 2,
"score": 5,
"scores_by_item": [
1,
1,
1,
0,
1,
0,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON string and interprets 300s retention and 30s visibility timeout as numeric seconds.",
"Identifies both R5 (short retention) and R6 (default visibility timeout) by substance.",
"No false positives about public access or misread wildcard policies; additional findings (no resource policy, SSE-SQS, DLQ polling) are legitimate observations, not fabricated misconfigs.",
"Labels both retention and visibility timeout as CRITICAL rather than treating them as soft/medium flags; the ground truth classifies these as medium severity soft flags, not confirmed critical bugs.",
"Acknowledges that visibility timeout correctness cannot be determined without knowing consumer p99 processing time, satisfying the boundary requirement.",
"Presents visibility timeout as a confirmed bug ('will be delivered to another consumer') rather than a flag to verify against actual consumer processing time; does not frame it conditionally as the ground truth requires.",
"Top fixes match ground truth in substance: raise retention period and set visibility timeout deliberately above consumer p99 processing time."
],
"agent_output": "## SQS Queue Audit: `click-events`\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 Message Retention Period is 5 minutes (source queue)\n\n**Attribute:** `MessageRetentionPeriod: \"300\"`\n\n**Problem:** Messages are permanently deleted after **300 seconds (5 minutes)**. For a high-volume analytics ingest queue, any consumer slowdown, deployment, incident, or backpressure event lasting more than 5 minutes will result in **silent, unrecoverable message loss**. The messages will not reach the DLQ \u2014 retention expiry is not a delivery failure, so the redrive policy does not apply. They are simply dropped.\n\n**Fix:** Set retention to at least `86400` (1 day), and ideally `345600`\u2013`604800` (4\u20137 days) to give you adequate incident response time.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes MessageRetentionPeriod=86400\n```\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 Visibility Timeout likely too short for real processing\n\n**Attribute:** `VisibilityTimeout: \"30\"`\n\n**Problem:** 30 seconds is the default and is frequently wrong. If your consumer takes longer than 30 seconds to process and acknowledge a message \u2014 due to downstream latency, batching, or any processing logic \u2014 the message becomes visible again and **will be delivered to another consumer**. This causes **duplicates**. Worse, each re-delivery increments the `ApproximateReceiveCount`. With `maxReceiveCount: 5`, a slow-but-succeeding consumer can exhaust the redrive threshold and push a successfully-processed message to the DLQ, causing **phantom DLQ alerts and apparent data loss**.\n\n**Fix:** Set the visibility timeout to **at least 6\u00d7 your p99 consumer processing time**. If processing takes up to 60 seconds, set this to 360+. If processing time is variable, use `ChangeMessageVisibility` to extend it dynamically during processing.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes VisibilityTimeout=360\n```\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 `maxReceiveCount: 5` may be too low given the visibility timeout risk above\n\n**Attribute (RedrivePolicy):** `maxReceiveCount: 5`\n\n**Problem:** With a 30-second visibility timeout and any processing latency, legitimate messages can be re-received multiple times without ever failing. A count of 5 is low enough that transient consumer issues (brief network blip, pod restart, cold start) will push messages to the DLQ **before they are ever successfully processed**. This is a data-loss risk disguised as an operational safety net.\n\n**Fix:** After correcting the visibility timeout, reassess this value. For analytics ingest where duplicates are tolerable but loss is not, a value of `10`\u2013`20` is more appropriate. This is a policy decision, but 5 is too aggressive at the current visibility timeout.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:click-events-dlq\\\",\\\"maxReceiveCount\\\":10}\"}'\n```\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 No queue access policy (resource policy) defined on either queue\n\n**Attribute:** `Policy` \u2014 **absent** on both queues\n\n**Problem:** Neither queue has an explicit resource-based policy. This means access is controlled **solely by IAM identity policies**. There is no defence-in-depth boundary preventing other principals in account `211125758836` with broadly-scoped IAM policies (e.g. `sqs:*` or `*`) from reading from or purging the source queue, or from sending arbitrary messages into it. For a data ingest queue, an unauthorized `SendMessage` would corrupt your analytics dataset silently. An unauthorized `PurgeQueue` or `ReceiveMessage` would cause unrecoverable loss.\n\n**Fix:** Add an explicit resource policy that **denies** access except from your known producer and consumer principal ARNs. At minimum, use a condition key to restrict to expected roles.\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowProducer\",\n \"Effect\": \"Allow\",\n \"Principal\": { \"AWS\": \"arn:aws:iam::211125758836:role/click-events-producer\" },\n \"Action\": \"sqs:SendMessage\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\"\n },\n {\n \"Sid\": \"AllowConsumer\",\n \"Effect\": \"Allow\",\n \"Principal\": { \"AWS\": \"arn:aws:iam::211125758836:role/click-events-consumer\" },\n \"Action\": [\"sqs:ReceiveMessage\", \"sqs:DeleteMessage\", \"sqs:ChangeMessageVisibility\", \"sqs:GetQueueAttributes\"],\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\"\n },\n {\n \"Sid\": \"DenyEverythingElse\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"sqs:*\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:click-events\",\n \"Condition\": {\n \"ArnNotLike\": {\n \"aws:PrincipalArn\": [\n \"arn:aws:iam::211125758836:role/click-events-producer\",\n \"arn:aws:iam::211125758836:role/click-events-consumer\"\n ]\n }\n }\n }\n ]\n}\n```\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 Encryption is SQS-managed (SSE-SQS), not KMS\n\n**Attribute:** `SqsManagedSseEnabled: \"true\"`, no `KmsMasterKeyId`\n\n**Problem:** SSE-SQS encrypts at rest but uses AWS-managed keys with **no customer control**. You cannot audit key usage via CloudTrail, cannot rotate on your schedule, cannot restrict decrypt access per-role, and cannot revoke access in an incident. For analytics click data that may contain user identifiers or behavioural data, this may violate your compliance posture (SOC2, GDPR, etc.).\n\n**Fix:** If your threat model or compliance requirements warrant it, migrate to SSE-KMS with a customer-managed key (CMK). This gives you full CloudTrail auditability of every `Decrypt` call and the ability to restrict or revoke key access independently.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes KmsMasterKeyId=arn:aws:kms:eu-west-1:211125758836:key/<your-cmk-id>\n```\nNote: ensure your consumer/producer IAM roles have `kms:Decrypt` / `kms:GenerateDataKey` on the CMK, or consumers will fail silently.\n\n---\n\n### \ud83d\udfe2 LOW \u2014 DLQ `ReceiveMessageWaitTimeSeconds` is 0 (short polling)\n\n**Attribute on DLQ:** `ReceiveMessageWaitTimeSeconds: \"0\"`\n\n**Problem:** The DLQ uses short polling. If you have automated DLQ monitoring or a redriver process that polls it, short polling will **miss messages on some requests** (SQS distributes messages across servers; short poll only samples a subset). This can cause your alerting to miss DLQ depth, or your redriver to fail to drain the queue. It also increases API call costs unnecessarily.\n\n**Fix:** Set to `20` to match the source queue, unless you have a specific reason to poll short.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events-dlq \\\n --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n### \ud83d\udfe2 LOW / OBSERVATION \u2014 ~1,190 messages currently in-flight or queued\n\n**Attribute:** `ApproximateNumberOfMessages: \"1190\"`\n\nThis is not a misconfiguration, but given the **5-minute retention period**, this is an **active emergency**. If these messages arrived recently and consumers are not keeping up, they may already be expiring. Treat this as a live incident until the retention period is corrected and queue depth is confirmed draining.\n\n---\n\n## Summary Table\n\n| Severity | Issue | Risk |\n|---|---|---|\n| \ud83d\udd34 Critical | Retention = 5 minutes | Silent message loss on any slowdown |\n| \ud83d\udd34 Critical | Visibility timeout = 30s (likely wrong) | Duplicates + false DLQ routing |\n| \ud83d\udfe1 Medium | maxReceiveCount = 5 (too low) | Legitimate messages sent to DLQ |\n| \ud83d\udfe1 Medium | No resource policy on either queue | Unauthorized access / data injection |\n| \ud83d\udfe1 Medium | SSE-SQS not SSE-KMS | No key audit trail or access revocation |\n| \ud83d\udfe2 Low | DLQ short polling | DLQ monitoring may miss messages |\n| \ud83d\udfe2 Observation | 1,190 messages queued with 5m retention | Possible active data loss right now |",
"elapsed_s": 54.7
},
{
"queue": "05-default-visibility-short-retention",
"condition": "treatment",
"trial": 0,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON, maxReceiveCount=5, deadLetterTargetArn, and all string-typed second values.",
"Identifies both R5 and R6 by substance, matching the ground-truth findings exactly.",
"No false positives introduced; the scoped RedriveAllowPolicy and SSE are correctly noted as clean.",
"R6 (silent message loss) is ranked MEDIUM and headlined first, above R5 (LOW soft flag), correctly prioritizing loss over double-delivery.",
"Explicitly names consumer processing time and CloudWatch metrics as things that cannot be determined from config alone, matching the ground-truth boundary.",
"R5 is presented as a flag to verify against consumer p99 processing time, not a confirmed bug; R6 is framed as a condition for potential loss, not confirmed ongoing loss.",
"Top fix recommends raising MessageRetentionPeriod to cover plausible outages and setting VisibilityTimeout deliberately above consumer p99, matching ground-truth fix in substance."
],
"agent_output": "# SQS Queue Audit: `click-events`\n\n---\n\n## Step 1: Parse the Attributes\n\n**Source queue \u2014 parsed values:**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `VisibilityTimeout` | `\"30\"` | 30 s |\n| `MessageRetentionPeriod` | `\"300\"` | 300 s (5 minutes) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `\"{...}\"` | `deadLetterTargetArn = arn:aws:sqs:eu-west-1:211125758836:click-events-dlq`, `maxReceiveCount = 5` |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n| `Policy` | absent | no resource policy |\n| `FifoQueue` | absent | standard queue |\n\n**Dead-letter queue \u2014 parsed values:**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `MessageRetentionPeriod` | `\"1209600\"` | 1,209,600 s (14 days) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n| `RedriveAllowPolicy` | `\"{...}\"` | `byQueue`, source restricted to `click-events` ARN |\n\n---\n\n## Step 2: Apply the Rules\n\n### R3 \u2014 DLQ retention vs source retention\n- Source retention: **300 s**\n- DLQ retention: **1,209,600 s**\n- 1,209,600 > 300 \u2713 \u2014 DLQ retention is longer. **R3 does not fire.**\n\n### R4 \u2014 Poison message age-out arithmetic\n- `maxReceiveCount \u00d7 VisibilityTimeout = 5 \u00d7 30 = 150 s`\n- `MessageRetentionPeriod = 300 s`\n- 150 s \u2264 300 s \u2713 \u2014 messages can exhaust their receive count within retention. **R4 does not fire.**\n\n### R1 \u2014 DLQ present?\n- `RedrivePolicy` is present, `deadLetterTargetArn` is populated. **R1 does not fire.**\n\n### R2 \u2014 `maxReceiveCount` band (3\u201310)?\n- `maxReceiveCount = 5`. Within band. **R2 does not fire.**\n\n### R5 \u2014 Visibility timeout at 30s default?\n- `VisibilityTimeout = 30`. Matches the default. **R5 fires (low).**\n\n### R6 \u2014 Retention shorter than a plausible outage?\n- `MessageRetentionPeriod = 300 s = 5 minutes`. Far below 1 hour. **R6 fires (medium).**\n\n### R7 \u2014 Open resource policy?\n- No `Policy` attribute present. **R7 does not fire.**\n\n### R8 \u2014 Encryption at rest disabled?\n- `SqsManagedSseEnabled = true`. **R8 does not fire.**\n\n### R9 \u2014 FIFO with content-based dedup off?\n- Standard queue, not FIFO. **R9 does not fire.**\n\n---\n\n## Output\n\n### Queue\n\n| | |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:click-events` |\n| **Type** | Standard |\n| **DLQ** | Wired \u2014 `arn:aws:sqs:eu-west-1:211125758836:click-events-dlq` |\n\n---\n\n### Findings\n\n#### \ud83d\udfe0 MEDIUM \u2014 R6: Retention shorter than a plausible outage\n\n**Attributes:** `MessageRetentionPeriod = 300` (5 minutes)\n\n**What breaks:** Five minutes is shorter than almost any operational gap that can occur in practice \u2014 a rolling deploy, a consumer crash, an ECS task rescheduling, a brief network partition. Any message that arrives while consumers are not processing and is not consumed within 5 minutes is deleted by SQS with no signal and no entry in the DLQ (expiry by age is not a receive event; it does not increment the receive count and does not trigger the redrive path). For a high-volume analytics ingest queue, this means a short consumer outage silently produces a gap in the click-event record that is invisible from the queue's own metrics after the fact.\n\n**Recommendation:** Raise `MessageRetentionPeriod` to cover at minimum the longest expected consumer downtime plus recovery headroom. For an analytics ingest queue where replay is valuable, 1 hour (3,600 s) is a practical floor; 24 hours (86,400 s) or more is common. Set it on the source queue; the DLQ's 14-day retention is already appropriate.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes MessageRetentionPeriod=86400\n```\n\n---\n\n#### \ud83d\udfe1 LOW \u2014 R5: Visibility timeout at the 30-second default\n\n**Attributes:** `VisibilityTimeout = 30`\n\n**What breaks:** If any consumer takes longer than 30 seconds to process and acknowledge a message, SQS makes that message visible again and delivers it a second time to a concurrent consumer. This is a double-processing risk, not a message-loss risk. Whether any consumer actually takes longer than 30 seconds is a property of the consumer, not the queue, so this cannot be confirmed from configuration alone. At high ingest volumes on an analytics queue, even occasional slow consumers (enrichment calls, batched writes, downstream latency spikes) can trigger this.\n\n**Recommendation:** Verify the p99 processing time for all consumers of this queue. If any consumer can exceed 30 seconds, raise `VisibilityTimeout` to at least 1.5\u00d7 the p99 processing time. If consumers use long-polling with `ReceiveMessageWaitTimeSeconds = 20` (as configured here), factor in that wait time at the tail. Consumers that extend their own visibility timeout via `ChangeMessageVisibility` can compensate dynamically, but the queue default should not require them to do so from the very first receive.\n\n---\n\n### Boundary\n\nThis audit is complete for what the static configuration of one queue can determine. The following questions are answerable only by crossing a join this audit cannot make:\n\n1. **Consumer processing time (R5 confirmation).** Whether the 30-second visibility timeout is actually too short depends on how long each consumer takes to process a message. That is a CloudWatch or application-level metric, not a queue attribute.\n\n2. **Live message-loss rate.** The 5-minute retention (R6) creates conditions for silent loss during any consumer gap, but whether loss is occurring *right now* \u2014 and how much \u2014 is visible only in `NumberOfMessagesSent` vs `NumberOfMessagesDeleted` in CloudWatch, and in the `ApproximateAgeOfOldestMessage` alarm. The 1,190 messages currently visible in `ApproximateNumberOfMessages` are a point-in-time count, not a trend.\n\n3. **DLQ drain ownership.** The DLQ currently holds 2 messages. The queue configuration confirms the redrive path is correctly wired, but it cannot confirm that anyone is monitoring the DLQ, alerting on it, or draining it. A DLQ that fills silently is functionally equivalent to no DLQ for operational purposes.\n\n4. **Effective access control.** There is no resource policy on this queue, so the only principals authorised to send or receive are those with identity policies granting access. The full set of those principals is the account's IAM graph, which is not visible here. The queue-side half of the access story is clean (no open policy); the identity-policy half is behind this boundary.\n\n5. **Producer behaviour.** Whether the right services are writing to this queue, at the expected volume, with correct message structure, is not a queue attribute. The high current depth (1,190 messages) against a 5-minute retention window is worth investigating: it may indicate consumers are behind, or that the queue is receiving an unexpected volume spike that, combined with R6, is already producing loss.",
"elapsed_s": 52.3
},
{
"queue": "05-default-visibility-short-retention",
"condition": "treatment",
"trial": 1,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON (deadLetterTargetArn, maxReceiveCount=5) and all string-typed seconds (VisibilityTimeout=30, MessageRetentionPeriod=300, DLQ=1209600).",
"Identifies both R5 (visibility timeout at 30s default) and R6 (retention at 300s) matching ground-truth findings exactly.",
"No false positives raised; correctly clears R1, R2, R3, R4, R7, R8, R9 with valid reasoning.",
"R6 (silent message loss / retention) is ranked MEDIUM and headlined first; R5 is ranked LOW and listed second, correctly prioritizing the loss risk.",
"Explicitly names consumer processing time, live queue metrics, IAM identity policies, producer behaviour, and DLQ monitoring as things not determinable from config alone.",
"R5 is explicitly presented as a risk flag dependent on consumer p99 processing time, not a confirmed bug, with clear conditional language throughout.",
"Top fix recommendation is to raise MessageRetentionPeriod (to 3600s or 86400s) and set VisibilityTimeout deliberately above consumer p99, matching the ground-truth fix in substance."
],
"agent_output": "# SQS Queue Audit: `click-events`\n\n---\n\n## Step 1: Parse the Attributes\n\n**Source queue parsed values:**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:click-events` |\n| `VisibilityTimeout` | `\"30\"` | **30 seconds** |\n| `MessageRetentionPeriod` | `\"300\"` | **300 seconds (5 minutes)** |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `\"{...}\"` | `deadLetterTargetArn = arn:aws:sqs:eu-west-1:211125758836:click-events-dlq`, `maxReceiveCount = 5` |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**DLQ parsed values:**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:click-events-dlq` |\n| `MessageRetentionPeriod` | `\"1209600\"` | **1,209,600 seconds (14 days)** |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n\n**Key arithmetic pre-computed before judgment:**\n\n- `maxReceiveCount \u00d7 VisibilityTimeout` = 5 \u00d7 30 s = **150 seconds**\n- Source `MessageRetentionPeriod` = **300 seconds**\n- DLQ `MessageRetentionPeriod` = **1,209,600 seconds**\n\n---\n\n## Step 2\u20135: Apply All Rules\n\n### R3 \u2014 DLQ retention vs source retention\nDLQ retention (1,209,600 s) is **greater than** source retention (300 s). \u2705 No R3 finding.\n\n### R4 \u2014 Poison messages age out before reaching the DLQ\n`maxReceiveCount \u00d7 VisibilityTimeout` = 150 s \n`MessageRetentionPeriod` = 300 s \n150 s \u2264 300 s \u2192 **the inequality does not fire**. \u2705 No R4 finding.\n\n*(Margin is thin \u2014 only 150 s of headroom \u2014 but the configured values satisfy the constraint, and the methodology instructs not to raise R4 speculatively.)*\n\n### R1 \u2014 No DLQ\n`RedrivePolicy` is present and points to a valid DLQ. \u2705 No R1 finding.\n\n### R2 \u2014 maxReceiveCount out of band\n`maxReceiveCount = 5`. Within the 3\u201310 band. \u2705 No R2 finding.\n\n### R6 \u2014 Retention shorter than a plausible outage\nSource retention = 300 seconds = **5 minutes**. A 5-minute consumer outage \u2014 a rolling deploy, a Lambda cold-start burst, a brief network blip \u2014 will cause messages queued during that window to expire before they are ever read. Triggers **R6 (medium)**.\n\n### R5 \u2014 Visibility timeout at the 30s default\n`VisibilityTimeout = 30`. Default value. Risk flag: if a consumer takes longer than 30 s, the message reappears and is delivered twice. Triggers **R5 (low)**.\n\n### R7 \u2014 Open resource policy\nNo `Policy` attribute present. \u2705 No R7 finding.\n\n### R8 \u2014 Encryption at rest disabled\n`SqsManagedSseEnabled = true`. Encryption is on. \u2705 No R8 finding.\n\n### R9 \u2014 FIFO with content-based dedup off\n`FifoQueue` absent \u2014 this is a standard queue. \u2705 No R9 finding.\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:click-events` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2192 `arn:aws:sqs:eu-west-1:211125758836:click-events-dlq` |\n| **Role** | High-volume analytics ingest |\n\n---\n\n### Findings\n\n#### \ud83d\udfe0 MEDIUM \u2014 R6: Retention shorter than a plausible outage\n\n**Rule:** R6 \n**Grounded in:** `MessageRetentionPeriod = 300` (5 minutes)\n\n**What breaks:** \nSQS retention is measured from the original `SentTimestamp`. Any message not received within 300 seconds of being sent is permanently deleted by SQS \u2014 silently, with no signal to the producer and no movement to the DLQ (retention expiry does not trigger the redrive path; it is a hard delete). For an analytics ingest queue, the failure modes include:\n\n- A rolling deployment of the consumer that pauses processing for longer than 5 minutes.\n- A Lambda concurrency limit or throttle that causes sustained receive lag.\n- Any on-call incident response that takes longer than 5 minutes to act on an alert.\n\nIn every one of these cases, click events in flight are deleted, not quarantined.\n\n**Recommendation:** \nRaise `MessageRetentionPeriod` to at minimum 3,600 seconds (1 hour), which provides headroom for a normal operational response. For an analytics queue where re-ingestion is non-trivial, 86,400 seconds (1 day) is a more defensible default and still leaves the DLQ's 14-day retention well ahead of it. Update via:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/click-events \\\n --attributes MessageRetentionPeriod=86400\n```\n\n---\n\n#### \ud83d\udd35 LOW \u2014 R5: Visibility timeout at the 30-second default\n\n**Rule:** R5 \n**Grounded in:** `VisibilityTimeout = 30`\n\n**What breaks (risk flag, not proven):** \nIf any consumer of this queue takes longer than 30 seconds to process a single click-event message, SQS will return the message as visible and deliver it to a second consumer before the first has finished. The result is duplicate processing \u2014 two writes of the same click event into whatever analytics store sits downstream. Whether this actually happens depends on consumer processing time, which is not visible in the queue configuration.\n\nFor a high-volume analytics ingest queue, consumers often batch-write to downstream stores (Kinesis Firehose, Redshift, S3 via Glue, etc.); batch-commit latency frequently exceeds 30 seconds under load.\n\n**Recommendation:** \nMeasure the p99 processing time of the consumer under production load. Set `VisibilityTimeout` to at least 6\u00d7 that value (to absorb retries and slow outliers). If the consumer uses `ChangeMessageVisibility` to extend the timeout mid-processing, verify that heartbeat interval is less than the current timeout. A common safe starting point for analytics consumers is 300 seconds; match it to your measurement, not a guess.\n\n---\n\n### \u26a0\ufe0f Additional note: R4 margin\n\nNo R4 finding is raised \u2014 the arithmetic is satisfied (`maxReceiveCount \u00d7 VisibilityTimeout = 150 s < MessageRetentionPeriod = 300 s`). However the safety margin is exactly 150 seconds. **If `MessageRetentionPeriod` is lowered without also lowering `maxReceiveCount \u00d7 VisibilityTimeout`, or if `VisibilityTimeout` is raised to address R5, the constraint can be violated.** For example: raising `VisibilityTimeout` to 300 s while keeping `maxReceiveCount = 5` would produce `5 \u00d7 300 = 1500 s > 300 s`, which would immediately trigger a critical R4 condition. The two remediations must be planned together:\n\n- If you raise `VisibilityTimeout` to fix R5, raise `MessageRetentionPeriod` (already recommended for R6) first, so the R4 product stays below the new retention value.\n- The safe sequencing is: **raise retention \u2192 then raise visibility timeout**.\n\n---\n\n### Boundary\n\nThis audit covers the static configuration of one queue and its DLQ. It cannot answer the following questions, and those gaps are real:\n\n1. **Consumer processing time.** Whether 30 seconds is long enough for R5 cannot be answered here. The visibility timeout is a queue attribute; the processing time is a property of the consumer service. R5 is a flag, not a verdict. Close it by measuring the p99 consumer duration in CloudWatch (`Duration` for Lambda, or application-level metrics for a long-running consumer).\n\n2. **Live queue behaviour.** `ApproximateNumberOfMessages = 1190` on the source queue and `ApproximateNumberOfMessages = 2` on the DLQ are point-in-time snapshots, not audit inputs \u2014 the methodology does not use them for findings. Whether messages are currently accumulating, whether the DLQ is receiving new arrivals, and what the age of the oldest message is require CloudWatch metrics (`ApproximateAgeOfOldestMessage`, `NumberOfMessagesSent`, `NumberOfMessagesDeleted`) correlated over time. The 1,190 queued messages are visible here but their significance \u2014 backlog, outage, or normal steady-state \u2014 cannot be determined from the attribute alone.\n\n3. **Account-level IAM.** No resource policy is attached to this queue, so the queue grants no permissions directly. However, identity policies attached to IAM roles, users, and services in account `211125758836` may grant `sqs:SendMessage`, `sqs:ReceiveMessage`, or `sqs:DeleteMessage` independently. The effective permission set is the union of the resource policy (here: none) and every identity policy in the account. That union is not readable from the queue.\n\n4. **Producers.** Which services are writing click events to this queue, at what rate, and whether any producer is retrying on error (which could cause the queue depth to grow faster than the 5-minute retention window can serve) are properties of the producers, not the queue. The current 5-minute retention makes producer behaviour particularly consequential: a producer retry storm that outpaces the consumer will cause message loss under the current retention, even after remediation is in progress.\n\n5. **DLQ consumer.** The DLQ currently holds 2 messages. Whether anyone is monitoring it, draining it, or alerting on it is not visible here. A DLQ with no consumer is a quarantine with no one checking the ward. Verify that a CloudWatch alarm exists on `ApproximateNumberOfMessagesVisible` for `click-events-dlq` with a threshold of 1.\n\n6. **Downstream analytics store.** Whether duplicate delivery (R5 risk) would cause a problem depends on whether the downstream store is idempotent (e.g., deduplication by event ID). That is a property of the downstream system, not this queue.",
"elapsed_s": 67.0
},
{
"queue": "05-default-visibility-short-retention",
"condition": "treatment",
"trial": 2,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON, maxReceiveCount=5, and all string-typed seconds including the 300s retention and 30s visibility timeout.",
"Identifies both R5 (visibility timeout at default) and R6 (short retention) matching the ground-truth findings exactly.",
"No false positives raised; absent resource policy is correctly noted as absence rather than a misconfiguration.",
"R6 (silent message loss) is ranked as MEDIUM and headlined first, while R5 is ranked LOW, correctly prioritizing the message-loss risk.",
"Boundary section explicitly names consumer processing time, live queue behavior/CloudWatch metrics, and IAM policies as things not determinable from config alone.",
"R5 is explicitly framed as conditional ('cannot be determined from the queue configuration alone') and R6 is presented as a soft flag requiring context about tolerable outage duration.",
"Top fix correctly recommends raising MessageRetentionPeriod (e.g., to 86400s or 345600s) and setting VisibilityTimeout above consumer p99 processing time, matching the ground-truth fix in substance."
],
"agent_output": "# SQS Queue Audit: click-events\n\n---\n\n## Step 1 \u2014 Parse the attributes\n\n**Source queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:click-events` |\n| `VisibilityTimeout` | `\"30\"` | 30 s |\n| `MessageRetentionPeriod` | `\"300\"` | 300 s (5 minutes) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | JSON string | `deadLetterTargetArn = arn:aws:sqs:eu-west-1:211125758836:click-events-dlq`, `maxReceiveCount = 5` |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n| `Policy` | absent | no resource policy |\n| `FifoQueue` | absent | standard queue |\n\n**Dead-letter queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:click-events-dlq` |\n| `MessageRetentionPeriod` | `\"1209600\"` | 1 209 600 s (14 days) |\n| `VisibilityTimeout` | `\"30\"` | 30 s |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n\nParsing complete. Proceeding to judgment.\n\n---\n\n## Step 2 \u2014 Audit the redrive path\n\n**R1 \u2014 No DLQ:** `RedrivePolicy` is present. DLQ is wired. R1 does not fire.\n\n**R2 \u2014 maxReceiveCount out of band:** `maxReceiveCount = 5`. This is within the 3\u201310 band. R2 does not fire.\n\n**R3 \u2014 DLQ retention not longer than source retention:**\n- Source retention: 300 s\n- DLQ retention: 1 209 600 s\n\nDLQ retention (14 days) is substantially longer than source retention (5 minutes). The ordering is correct. R3 does not fire.\n\n---\n\n## Step 3 \u2014 Audit the message lifecycle\n\n**R4 \u2014 Poison messages age out before reaching the DLQ:**\n\nThe arithmetic:\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout = 5 \u00d7 30 s = 150 s\nMessageRetentionPeriod = 300 s\n```\n\n`150 s < 300 s` \u2014 the inequality is *not* violated. A poison message needs at most 150 s of wall-clock to exhaust its receive count; retention does not expire first. R4 does not fire.\n\n**R5 \u2014 Visibility timeout at the 30 s default:**\n\n`VisibilityTimeout = 30 s` \u2014 this is the AWS default. Any consumer that takes longer than 30 s to process a click-event will cause the message to reappear and be delivered to a second consumer (or the same one again), producing a duplicate delivery. Whether this queue's consumers exceed 30 s is not determinable from configuration alone. **R5 fires (low).**\n\n**R6 \u2014 Retention shorter than a plausible outage:**\n\n`MessageRetentionPeriod = 300 s` \u2014 five minutes. A deploy, a brief consumer crash, an autoscaling warmup, or a Lambda cold-start cascade need only outlast five minutes to silently drop every message that arrived during the gap. For a high-volume analytics ingest queue this means silent data loss under any routine operational event. **R6 fires (medium).**\n\n---\n\n## Step 4 \u2014 Audit exposure\n\n**R7 \u2014 Open resource policy:** No `Policy` attribute is present. R7 does not fire.\n\n**R8 \u2014 Encryption at rest disabled:** `SqsManagedSseEnabled = true` on both source and DLQ. R8 does not fire.\n\n---\n\n## Step 5 \u2014 Audit FIFO invariants\n\n`FifoQueue` attribute is absent \u2014 this is a standard queue. R9 does not apply.\n\n---\n\n## Step 6 \u2014 Ranked findings and boundary\n\n---\n\n# Audit Report\n\n## Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:click-events` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2192 `arn:aws:sqs:eu-west-1:211125758836:click-events-dlq` |\n| **maxReceiveCount** | 5 |\n| **Encryption** | SQS-managed SSE (both queues) |\n\n---\n\n## Findings\n\n### \ud83d\udd34 MEDIUM \u2014 R6: Retention shorter than a plausible outage\n\n**Attributes:** `MessageRetentionPeriod = 300` (5 minutes)\n\n**What breaks:** SQS deletes any message that has been in the queue for longer than the retention period, regardless of whether it has been consumed. Five minutes is shorter than many ordinary operational gaps: a rolling deploy, a consumer crash-loop, an autoscaling warmup, a transient downstream dependency failure. Any such event that outlasts 300 s causes every message queued during that window to be permanently and silently deleted \u2014 they will never reach the consumer and they will never reach the DLQ (the DLQ only receives messages that have been received and failed, not messages that expired while waiting). For a high-volume analytics ingest queue this is a silent data-loss path under routine conditions.\n\n**Recommendation:** Raise `MessageRetentionPeriod` to a value that comfortably exceeds the worst-case operational gap you are willing to tolerate. For an analytics ingest queue, 86 400 s (1 day) is a conventional floor; 345 600 s (4 days) gives comfortable incident headroom. The DLQ's 14-day retention is already well-sized; align the source to it.\n\n---\n\n### \ud83d\udfe1 LOW \u2014 R5: Visibility timeout at the 30 s default\n\n**Attributes:** `VisibilityTimeout = 30` (source queue)\n\n**What breaks:** When a consumer receives a message, the message is hidden from other consumers for exactly 30 s. If the consumer has not deleted the message (i.e. finished processing) before that window expires, SQS makes the message visible again and it is re-delivered \u2014 to the same consumer or a different one. This increments the receive count toward `maxReceiveCount = 5`. A consumer that consistently takes longer than 30 s will therefore cause every message to be delivered multiple times, and those messages will eventually dead-letter not because they failed but because the visibility window was too short. Whether this queue's consumers exceed 30 s cannot be determined from the queue configuration alone.\n\n**Recommendation:** Measure the 99th-percentile processing time for this queue's consumers (CloudWatch `ApproximateAgeOfOldestMessage`, application timing logs, or Lambda duration metrics). Set `VisibilityTimeout` to at least 1.5\u00d7 that value, with a minimum of 60 s for any non-trivial consumer. If consumers sometimes extend their own visibility timeout in-flight, verify that the extension logic is working.\n\n---\n\n### \u2139\ufe0f No further findings\n\nR1, R2, R3, R4, R7, R8, R9 do not fire. The redrive path is correctly wired and arithmetically sound (a poison message needs at most 150 s to exhaust its receive count; retention is 300 s). Both queues carry SQS-managed SSE. The resource policy is absent. `maxReceiveCount = 5` is within the safe band. DLQ retention (14 days) exceeds source retention (5 minutes) by a large margin.\n\n---\n\n## Boundary\n\nThis audit read the static configuration of `click-events` and its DLQ. It cannot answer the following questions, and they are the ones most likely to matter next:\n\n1. **Consumer processing time (R5 confirmation).** Whether 30 s is long enough for this queue's consumers is a property of the consumer, not the queue. The fix for R5 requires Lambda duration percentiles, ECS task timing, or equivalent application metrics \u2014 none of which are in `GetQueueAttributes`.\n\n2. **Live queue behaviour.** `ApproximateNumberOfMessages = 1190` is visible in the attributes snapshot. Whether that backlog is growing, stable, or draining; what the age of the oldest message is relative to the 300 s retention window; and whether messages are already expiring before consumption \u2014 all of these require CloudWatch (`NumberOfMessagesDeleted`, `ApproximateAgeOfOldestMessage`, `NumberOfMessagesSent` vs `NumberOfMessagesDeleted` delta). A 5-minute retention window with 1 190 messages currently queued is a configuration under pressure; the live metrics would tell you whether it is already losing messages.\n\n3. **The effective access-control boundary.** No resource policy is attached to this queue, so access is governed entirely by IAM identity policies in account `211125758836`. The audit sees none of those. The full set of principals that can send to, receive from, or purge this queue is the union of the (absent) resource policy and every identity policy in the account \u2014 visible only from IAM.\n\n4. **Producer behaviour and message volume.** Whether the right services are writing to this queue, at what rate, and whether any producer is sending malformed messages that will consistently fail and inflate the receive count \u2014 none of this is in the queue's attributes.\n\n5. **DLQ drainage.** The DLQ currently holds 2 messages. Whether anyone is monitoring it, alerting on it, and consuming from it (to replay or discard failed events) is a property of the system around the queue, not the queue itself. A DLQ that is never drained becomes a silent graveyard.\n\n6. **R6 impact under current load.** The recommendation to raise retention is derivable from configuration alone, but the *urgency* depends on whether messages are already expiring. With 1 190 messages in-flight and a 5-minute retention period, CloudWatch `ApproximateAgeOfOldestMessage` is the immediate operational check to run.",
"elapsed_s": 65.4
},
{
"queue": "08-clean-standard",
"condition": "control",
"trial": 0,
"score": 1,
"scores_by_item": [
1,
0,
0,
0,
0,
0,
0
],
"notes": [
"Agent correctly parses the RedrivePolicy JSON string and numeric attributes like VisibilityTimeout and MessageRetentionPeriod.",
"Ground truth has zero findings; agent invents 7 findings, failing to recognize this as a clean queue.",
"Agent explicitly calls the aws:SourceArn-scoped wildcard principal a 'MEDIUM' security finding, which the ground truth explicitly says is the legitimate SNS-to-SQS pattern and not a defect.",
"Agent labels invented findings as HIGH severity, inverting the correct answer of no findings at all.",
"Agent does not name any of the ground-truth boundaries (metrics over time, IAM graph, DLQ-to-owner relationship) as things it cannot determine.",
"Agent presents soft flags like VisibilityTimeout mismatch and maxReceiveCount as confirmed bugs rather than hypotheses to verify against actual workload behavior.",
"Ground truth says the correct top fix is 'None'; agent recommends multiple remediation changes on a correctly configured queue."
],
"agent_output": "## SQS Queue Audit: `notification-fanout`\n\n---\n\n### FINDING 1 \u2014 `maxReceiveCount` Too Low\n**Severity: HIGH**\n\n**Location:** `RedrivePolicy.maxReceiveCount = 5`\n\n**Problem:**\nWith a `VisibilityTimeout` of 180 seconds, a message gets 5 attempts before being moved to the DLQ. If your workers experience transient failures (brief downstream outage, cold start, throttling), 5 attempts is easily exhausted, causing legitimate messages to be silently routed to the DLQ rather than retried to success. This is message loss from the processing perspective \u2014 the worker never successfully handles it.\n\nThis is especially risky because 5 attempts \u00d7 180s = 15 minutes of retry window before permanent DLQ routing. For a notification fanout, transient failures longer than that result in dropped notifications.\n\n**Fix:**\nRaise `maxReceiveCount` to at least `10`\u2013`20` depending on acceptable retry duration. Calibrate against your workers' observed error patterns and the SNS message delivery SLA.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout \\\n --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq\\\",\\\"maxReceiveCount\\\":\\\"15\\\"}\"}'\n```\n\n---\n\n### FINDING 2 \u2014 Message Retention Period Too Short on Source Queue\n**Severity: HIGH**\n\n**Location:** `MessageRetentionPeriod = 345600` (4 days)\n\n**Problem:**\nIf the worker fleet goes down for a weekend (e.g., a deployment failure, an ASG misconfiguration, a prolonged incident) messages start expiring after 4 days. SQS **silently deletes** expired messages \u2014 there is no DLQ fallback for expiry, the redrive policy does not apply. Those notifications are permanently lost with no visibility.\n\nThe DLQ already has 14 days (`1209600`), so the operational intent appears to be long retention, but the source queue doesn't match that intent.\n\n**Fix:**\nAlign the source queue retention to at least 7 days, ideally 14 to match the DLQ:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout \\\n --attributes '{\"MessageRetentionPeriod\":\"1209600\"}'\n```\n\n---\n\n### FINDING 3 \u2014 `VisibilityTimeout` Mismatch Risk Against Worker Processing Time\n**Severity: MEDIUM**\n\n**Location:** `VisibilityTimeout = 180` (source and DLQ both)\n\n**Problem:**\n180 seconds is the configured timeout, but this is only safe if **every worker is guaranteed to complete processing and call `DeleteMessage` within 180 seconds**. If any worker task (e.g., a downstream HTTP call, a database write) can exceed 180 seconds:\n\n- The message becomes visible again while still being processed by the original worker\n- A second worker picks it up\n- Both process it simultaneously \u2192 **duplicate delivery and duplicate side effects** (duplicate notifications sent to users)\n\nThere is no evidence of `ChangeMessageVisibility` heartbeating in the configuration, which is the only mitigation if processing time is variable.\n\n**Fix:**\nEither:\n1. Set `VisibilityTimeout` to `max(expected processing time) \u00d7 1.5` with a hard ceiling enforced at the application layer, **or**\n2. Implement `ChangeMessageVisibility` heartbeating in the worker to extend the timeout for long-running tasks before it expires.\n\nIf your p99 processing time is, say, 60 seconds, 180 is fine. If it can reach 200+ seconds, you have a duplication bug.\n\n---\n\n### FINDING 4 \u2014 Queue Policy Uses Wildcard Principal\n**Severity: MEDIUM**\n\n**Location:** `Policy.Statement[0].Principal = \"*\"`\n\n**Problem:**\nThe policy correctly scopes to the SNS topic via `aws:SourceArn`, which does prevent arbitrary SNS topics from publishing. However, using `Principal: \"*\"` with a condition is a weaker pattern than explicitly scoping to the SNS service principal (`sns.amazonaws.com`). \n\nMore importantly: the `aws:SourceArn` condition only applies to the `sqs:SendMessage` action from AWS services that set the source ARN. If the condition evaluation has a bug or the policy is copy-pasted and the condition accidentally removed in a future update, `Principal: \"*\"` becomes an open queue accepting messages from anyone on the internet.\n\n**Fix:**\nScope the principal to the SNS service explicitly:\n\n```json\n\"Principal\": {\n \"Service\": \"sns.amazonaws.com\"\n},\n\"Condition\": {\n \"ArnEquals\": {\n \"aws:SourceArn\": \"arn:aws:sns:eu-west-1:211125758836:account-events\"\n }\n}\n```\n\nThis is defense-in-depth: two independent constraints must both be satisfied.\n\n---\n\n### FINDING 5 \u2014 No CloudWatch Alarm on DLQ Depth\n**Severity: MEDIUM**\n\n**Location:** `ApproximateNumberOfMessages = 1` on DLQ (observed at audit time)\n\n**Problem:**\nThere is already 1 message in the DLQ. This is not a configuration attribute, but the absence of an alarm on `ApproximateNumberOfMessages > 0` for the DLQ is a configuration gap. DLQ accumulation is silent by default. If this grows, it means messages are failing processing and **notifications are not being delivered**, with no one being paged.\n\nThe 14-day retention gives you time to react, but only if you know there's a problem.\n\n**Fix:**\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"notification-fanout-dlq-not-empty\" \\\n --metric-name ApproximateNumberOfMessages \\\n --namespace AWS/SQS \\\n --dimensions Name=QueueName,Value=notification-fanout-dlq \\\n --statistic Maximum \\\n --period 60 \\\n --threshold 1 \\\n --comparison-operator GreaterThanOrEqualToThreshold \\\n --evaluation-periods 1 \\\n --alarm-actions <your-sns-alert-arn> \\\n --treat-missing-data notBreaching\n```\n\n---\n\n### FINDING 6 \u2014 No Alarm on Source Queue Age (`ApproximateAgeOfOldestMessage`)\n**Severity: MEDIUM**\n\n**Location:** Source queue \u2014 no alarm configuration visible\n\n**Problem:**\n`ApproximateNumberOfMessages = 14` messages are sitting in the source queue right now. Without an alarm on message age, a stalled consumer fleet (crashed workers, scaling failure, IAM permission regression) will go undetected until the 4-day retention window starts causing silent drops (compounding Finding 2). Queue depth alone is a noisy signal; **message age is the correct leading indicator of a stuck consumer**.\n\n**Fix:**\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"notification-fanout-consumer-lag\" \\\n --metric-name ApproximateAgeOfOldestMessage \\\n --namespace AWS/SQS \\\n --dimensions Name=QueueName,Value=notification-fanout \\\n --statistic Maximum \\\n --period 300 \\\n --threshold 3600 \\\n --comparison-operator GreaterThanThreshold \\\n --evaluation-periods 2 \\\n --alarm-actions <your-sns-alert-arn>\n```\n\nThreshold should be calibrated to your expected processing SLA (1 hour shown above is a starting point).\n\n---\n\n### FINDING 7 \u2014 DLQ Has No Redrive (No DLQ-of-DLQ)\n**Severity: LOW**\n\n**Location:** DLQ has no `RedrivePolicy` configured\n\n**Problem:**\nIf a worker is mistakenly pointed at the DLQ for reprocessing and fails repeatedly, messages in the DLQ will be received multiple times. With no redrive on the DLQ, after 14 days of retention those messages are permanently deleted with no further fallback. For a notification system this may be acceptable, but if the DLQ is being actively consumed, poison-pill messages in the DLQ can cause consumer loops.\n\n**Fix:**\nEither accept this as a known and documented risk, or create a `notification-fanout-dlq-dlq` with a small `maxReceiveCount` (e.g., 3) for truly unprocessable messages, and set an alarm on it.\n\n---\n\n### Summary Table\n\n| # | Finding | Severity | Risk |\n|---|---------|----------|------|\n| 1 | `maxReceiveCount` too low (5) | HIGH | Messages prematurely DLQ'd, effective drop |\n| 2 | Source queue retention only 4 days | HIGH | Silent message expiry during outage |\n| 3 | VisibilityTimeout vs. actual processing time unvalidated | MEDIUM | Duplicate delivery / double notifications |\n| 4 | `Principal: \"*\"` in queue policy | MEDIUM | Over-permissive, fragile security posture |\n| 5 | No DLQ depth alarm (DLQ already has 1 message) | MEDIUM | Silent notification delivery failure |\n| 6 | No `ApproximateAgeOfOldestMessage` alarm | MEDIUM | Stalled consumer goes undetected |\n| 7 | No DLQ-of-DLQ | LOW | Poison pills consume DLQ retention silently |",
"elapsed_s": 54.4
},
{
"queue": "08-clean-standard",
"condition": "control",
"trial": 1,
"score": 1,
"scores_by_item": [
1,
0,
0,
0,
0,
0,
0
],
"notes": [
"Agent correctly parses RedrivePolicy and string-typed numeric attributes (maxReceiveCount, retention periods, visibility timeouts).",
"Ground truth has zero findings; agent invents six findings where none exist \u2014 fails to recognize the queue as clean.",
"Agent explicitly flags the aws:SourceArn-scoped wildcard principal as a 'MEDIUM' misconfiguration, which the ground truth calls a correctly narrowed legitimate SNS-to-SQS pattern \u2014 a direct false positive.",
"Agent headlines a HIGH finding about maxReceiveCount/VisibilityTimeout coupling on a clean queue rather than reporting no findings; criticality ordering is irrelevant when all findings are fabricated.",
"Agent does not name any boundary it cannot cross (metrics over time, IAM graph, DLQ ownership) \u2014 no boundary statement appears anywhere in the output.",
"Agent presents all six flags as confirmed bugs with explicit fix instructions rather than as hypothetical concerns requiring verification against actual consumer behavior or data classification.",
"Ground truth top fix is 'None'; agent's top fix is to raise maxReceiveCount or implement exponential backoff \u2014 directly contradicts the reference answer."
],
"agent_output": "## SQS Queue Audit: `notification-fanout`\n\n---\n\n### FINDING 1 \u2014 `maxReceiveCount` Too Low for `VisibilityTimeout` of 180s\n**Severity: HIGH**\n\n**Problem:**\n`maxReceiveCount: 5` means a message is dead-lettered after 5 failed processing attempts. With `VisibilityTimeout: 180s`, the total time before a message is permanently routed to the DLQ is only **15 minutes** (5 \u00d7 180s, assuming back-to-back failures with no backoff). If your workers experience transient infrastructure issues (AZ disruption, deployment rollout, cold-start spike) lasting longer than that window, legitimate messages get dead-lettered rather than retried through recovery.\n\nThe deeper issue is that `maxReceiveCount` and `VisibilityTimeout` are coupled but configured independently here with no apparent coordination.\n\n**Fix:**\nEither raise `maxReceiveCount` (10\u201315 is a common baseline for transient-fault tolerance), or implement exponential backoff in the worker by explicitly calling `ChangeMessageVisibility` to extend the timeout on retry. Define your retry budget intentionally: decide the maximum tolerable processing latency, then size `maxReceiveCount \u00d7 VisibilityTimeout` to comfortably exceed it.\n\n---\n\n### FINDING 2 \u2014 `MessageRetentionPeriod` of 4 Days on Source Queue Is Likely Too Short\n**Severity: MEDIUM**\n\n**Problem:**\nThe source queue retains messages for `345600` seconds (4 days). If the worker fleet goes down entirely (or falls severely behind) and isn't recovered within 4 days, messages are **silently dropped by SQS** \u2014 they do not get moved to the DLQ; they are simply deleted. This is a common operational blindspot. The DLQ correctly has 14-day retention (`1209600`), but that only helps messages that were actually dead-lettered before the source queue dropped them.\n\n**Fix:**\nIncrease `MessageRetentionPeriod` on the source queue to `1209600` (14 days, the SQS maximum) unless there is a deliberate business reason for a shorter window. This gives you maximum recovery time. Pair this with a CloudWatch alarm on `ApproximateAgeOfOldestMessage` to alert well before the retention cliff.\n\n---\n\n### FINDING 3 \u2014 No Alarm or Visibility on DLQ Depth (Operational Gap With a Message Already Present)\n**Severity: MEDIUM**\n\n**Problem:**\n`ApproximateNumberOfMessages: 1` on the DLQ right now \u2014 there is an actively dead-lettered message and no evidence of alerting configured at the queue level. SQS does not notify you when messages land in a DLQ; you must configure this explicitly. Without it, failed messages silently accumulate until the DLQ itself overflows or someone notices manually.\n\nThis is not surfaced in `get-queue-attributes`, but its absence is a confirmed operational misconfiguration given the current DLQ state.\n\n**Fix:**\nCreate a CloudWatch alarm:\n```\nMetric: AWS/SQS \u2013 ApproximateNumberOfMessagesVisible\nQueue: notification-fanout-dlq\nThreshold: >= 1\nPeriod: 60s\nAction: SNS alert to on-call\n```\nAdditionally, alarm on `ApproximateAgeOfOldestMessage` on the **source** queue with a threshold well below its retention period (e.g., alert at 1 day, hard-page at 2 days).\n\n---\n\n### FINDING 4 \u2014 Source Queue Policy Uses `\"Principal\": \"*\"`\n**Severity: MEDIUM**\n\n**Problem:**\nThe queue policy grants `sqs:SendMessage` to `Principal: \"*\"` (any AWS principal, any account), constrained only by a `Condition` on `aws:SourceArn`. This pattern is **weaker than it appears**. The `ArnEquals` condition on `aws:SourceArn` does restrict the SNS topic, but `aws:SourceArn` is only reliably set by AWS service principals (SNS, S3, EventBridge, etc.) \u2014 it is not a substitute for scoping the principal itself.\n\nA misconfigured or compromised IAM principal in *any* account that somehow bypasses the condition check (e.g., policy evaluation bug, future IAM path that doesn't set SourceArn) can write to this queue. The blast radius is uncontrolled.\n\n**Fix:**\nRestrict `Principal` to the SNS service principal and scope it to your account:\n```json\n\"Principal\": {\n \"Service\": \"sns.amazonaws.com\"\n},\n\"Condition\": {\n \"ArnEquals\": {\n \"aws:SourceArn\": \"arn:aws:sns:eu-west-1:211125758836:account-events\"\n }\n}\n```\nThis enforces defense-in-depth: only SNS as a service can invoke `SendMessage`, and only from that specific topic ARN. Both conditions must hold.\n\n---\n\n### FINDING 5 \u2014 `VisibilityTimeout` Mismatch Risk Between Queue and Consumer\n**Severity: LOW\u2013MEDIUM (context-dependent)**\n\n**Problem:**\n`VisibilityTimeout: 180s` is set at the queue level. If any worker takes longer than 180 seconds to process a message (e.g., downstream API latency, large payload processing), SQS will make the message visible again before the worker acknowledges it. This causes **duplicate processing** \u2014 another worker picks up the same message while the first is still working on it. With `VisibilityTimeout` identical on both source queue and DLQ (both `180`), this pattern is likely a copy-paste default rather than a deliberate design.\n\n**Fix:**\nAudit your p99 processing time for this worker fleet. `VisibilityTimeout` should be set to at least **2\u00d7 your p99 processing time** to absorb variance. If processing time is variable, workers should proactively call `ChangeMessageVisibility` to heartbeat the timeout before it expires. Ensure your consumers are idempotent regardless, as SQS is at-least-once delivery and duplicates can occur even with correct timeout configuration.\n\n---\n\n### FINDING 6 \u2014 No KMS CMK; Using SSE-SQS (`SqsManagedSseEnabled`)\n**Severity: LOW (risk-profile dependent)**\n\n**Problem:**\nBoth queues use `SqsManagedSseEnabled: true`, which is SQS-managed encryption. AWS manages the key; you have no control over key rotation policy, no ability to audit key usage via CloudTrail, no ability to revoke access by disabling a key, and no key policy to enforce access boundaries. For a queue handling account-level events (the SNS topic is `account-events`), this payload likely carries sensitive user or system data.\n\n**Fix:**\nIf this queue carries PII, credentials, or security-sensitive events, replace with a KMS CMK:\n```\n\"KmsMasterKeyId\": \"arn:aws:kms:eu-west-1:211125758836:key/<key-id>\"\n```\nEnsure the SNS topic's execution role and the worker fleet's role are granted `kms:GenerateDataKey` and `kms:Decrypt` respectively. Apply the same CMK to the DLQ, or a separate CMK with equivalent policy.\n\n---\n\n### Summary Table\n\n| # | Finding | Severity | Impact |\n|---|---------|----------|--------|\n| 1 | `maxReceiveCount: 5` too aggressive for 180s visibility timeout | HIGH | Premature dead-lettering under transient failures |\n| 2 | Source queue retention only 4 days vs. 14-day DLQ | MEDIUM | Silent message drop during extended outage |\n| 3 | No DLQ depth alarm; DLQ already has a message | MEDIUM | Silent failure accumulation, missed dead letters |\n| 4 | Queue policy `Principal: \"*\"` overly broad | MEDIUM | Unscoped write access, weakened access control |\n| 5 | Visibility timeout may cause duplicates if processing exceeds 180s | LOW\u2013MED | Duplicate delivery under slow consumers |\n| 6 | SSE-SQS instead of CMK for sensitive event topic | LOW | No key auditability or revocation control |",
"elapsed_s": 52.0
},
{
"queue": "08-clean-standard",
"condition": "control",
"trial": 2,
"score": 2,
"scores_by_item": [
1,
0,
0,
0,
1,
0,
0
],
"notes": [
"Agent correctly parses the RedrivePolicy JSON and string-typed numeric attributes like VisibilityTimeout and MessageRetentionPeriod.",
"Ground truth is zero findings (clean queue); agent invents 7 findings, failing to report the correct result of no defects.",
"Agent explicitly flags the aws:SourceArn-scoped wildcard Principal as a 'MEDIUM' misconfiguration, which the ground truth states is the legitimate SNS-to-SQS pattern and not a defect.",
"Agent headlines invented HIGH findings rather than correctly reporting no findings; no silent-message-loss defect exists per ground truth.",
"Agent mentions it cannot determine things from config alone (e.g., worker processing times, consumer behavior), which partially aligns with the boundary concept.",
"Agent presents speculative issues like VisibilityTimeout mismatch, consumer lag, and maxReceiveCount as confirmed or near-confirmed bugs rather than flags to verify.",
"Ground truth top fix is 'None'; agent's top fix is raising maxReceiveCount to 15, which contradicts the correct answer."
],
"agent_output": "## SQS Queue Audit: `notification-fanout`\n\n---\n\n### ISSUE 1: `maxReceiveCount` Too Low\n**Severity: HIGH**\n\n**Finding:**\n```json\n\"RedrivePolicy\": {\"maxReceiveCount\": 5}\n```\nWith a `VisibilityTimeout` of 180 seconds, a worker has 3 minutes per attempt. Five attempts is a reasonable number on the surface, but the real risk is transient infrastructure failures (deployment rollouts, AZ hiccups, worker crashes mid-processing) causing legitimate messages to burn through retries and land in the DLQ prematurely, before any worker ever successfully processes them.\n\n**Fix:**\nSet `maxReceiveCount` to at least **10**, ideally **15\u201320** for a notification fanout where some downstream targets may be temporarily unavailable. Tune based on your p99 processing time and acceptable redelivery window.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout \\\n --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq\\\",\\\"maxReceiveCount\\\":\\\"15\\\"}\"}'\n```\n\n---\n\n### ISSUE 2: Message Retention Too Short on the Source Queue\n**Severity: HIGH**\n\n**Finding:**\n```json\n\"MessageRetentionPeriod\": \"345600\" // 4 days\n```\nIf the worker fleet goes down for an extended period (incident, bad deploy, capacity event) messages that haven't yet been DLQ'd will be **permanently deleted** after 4 days with no recovery path. This is silent, irreversible message loss.\n\n**Fix:**\nIncrease to **1209600 (14 days)** \u2014 the maximum \u2014 to match the DLQ retention and give you the full recovery window. There is no cost penalty meaningful enough to justify the data loss risk.\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout \\\n --attributes '{\"MessageRetentionPeriod\":\"1209600\"}'\n```\n\n---\n\n### ISSUE 3: `VisibilityTimeout` Mismatch Risk Against Worker Processing Time\n**Severity: MEDIUM**\n\n**Finding:**\n```json\n\"VisibilityTimeout\": \"180\"\n```\n180 seconds is set on both queues, which is consistent. However, this is a hard ceiling: if a worker takes longer than 3 minutes to process a message (e.g., a slow downstream HTTP call, a retry loop, GC pause), the message becomes visible again and a **second worker picks it up**, causing duplicate processing. There is no evidence from the config that workers call `ChangeMessageVisibility` to extend the timeout during long processing.\n\n**Fix:**\nEither:\n- **Raise** `VisibilityTimeout` to match your actual p99 + buffer (e.g., if processing can take 2 minutes, set 360\u2013600 seconds), or\n- **Require** workers to call `ChangeMessageVisibility` heartbeat during processing if duration is variable.\n\nValidate your actual worker processing time against this value before changing.\n\n---\n\n### ISSUE 4: `Principal: \"*\"` in Queue Policy\n**Severity: MEDIUM**\n\n**Finding:**\n```json\n{\n \"Effect\": \"Allow\",\n \"Principal\": \"*\",\n \"Action\": \"sqs:SendMessage\",\n \"Condition\": {\n \"ArnEquals\": {\"aws:SourceArn\": \"arn:aws:sns:eu-west-1:211125758836:account-events\"}\n }\n}\n```\nThe `Principal: \"*\"` allows **any AWS principal in the world** to send messages, constrained only by the `Condition`. The condition is correctly scoped to the SNS topic ARN, which does limit the practical attack surface. However, `aws:SourceArn` is evaluated against the caller's declared source, and using `Principal: \"*\"` means you are relying entirely on that condition as your only guard. If that condition were accidentally removed in a future policy update, the queue becomes publicly writable.\n\n**Fix:**\nScope `Principal` to your own account or, better, to the SNS service principal explicitly:\n\n```json\n{\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Service\": \"sns.amazonaws.com\"\n },\n \"Action\": \"sqs:SendMessage\",\n \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:notification-fanout\",\n \"Condition\": {\n \"ArnEquals\": {\n \"aws:SourceArn\": \"arn:aws:sns:eu-west-1:211125758836:account-events\"\n }\n }\n}\n```\n\n---\n\n### ISSUE 5: No DLQ Alarm \u2014 Operational Gap\n**Severity: MEDIUM**\n\n**Finding:**\n```json\n\"ApproximateNumberOfMessages\": \"1\" // DLQ already has a message\n```\nThere is already a message in the DLQ right now. There is no CloudWatch alarm visible in this configuration. This means failed messages are silently accumulating with no on-call notification.\n\n**Fix:**\nCreate a CloudWatch alarm on `ApproximateNumberOfMessagesVisible` for the DLQ:\n\n```bash\naws cloudwatch put-metric-alarm \\\n --alarm-name \"notification-fanout-dlq-not-empty\" \\\n --metric-name ApproximateNumberOfMessagesVisible \\\n --namespace AWS/SQS \\\n --dimensions Name=QueueName,Value=notification-fanout-dlq \\\n --statistic Sum \\\n --period 60 \\\n --threshold 1 \\\n --comparison-operator GreaterThanOrEqualToThreshold \\\n --evaluation-periods 1 \\\n --alarm-actions arn:aws:sns:eu-west-1:211125758836:ops-alerts\n```\n\n---\n\n### ISSUE 6: `ApproximateNumberOfMessages: 14` on Source Queue \u2014 Possible Consumer Lag\n**Severity: LOW / INVESTIGATIVE**\n\n**Finding:**\n```json\n\"ApproximateNumberOfMessages\": \"14\"\n```\n14 messages are sitting in the source queue. On its own this is not a misconfiguration, but combined with a 4-day retention window and no alarm on source queue depth, this could be the early sign of a stalled consumer fleet that will go undetected until messages expire.\n\n**Fix:**\nAdd a CloudWatch alarm on the source queue depth with a threshold appropriate for your expected throughput, and separately alarm on `ApproximateAgeOfOldestMessage` to catch processing stalls before retention expiry.\n\n---\n\n### ISSUE 7: DLQ `ReceiveMessageWaitTimeSeconds: 0` \u2014 Minor Inefficiency\n**Severity: LOW**\n\n**Finding:**\n```json\n\"ReceiveMessageWaitTimeSeconds\": \"0\" // DLQ\n```\nThe DLQ uses short polling. If any automated process polls the DLQ (e.g., a Lambda for alerting or redriving), it will burn API calls and potentially miss messages that aren't returned in short-poll samples. This is a cost and reliability concern, not a correctness one.\n\n**Fix:**\nSet long polling on the DLQ for consistency:\n\n```bash\naws sqs set-queue-attributes \\\n --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout-dlq \\\n --attributes '{\"ReceiveMessageWaitTimeSeconds\":\"20\"}'\n```\n\n---\n\n### Summary Table\n\n| # | Issue | Risk | Severity |\n|---|-------|------|----------|\n| 1 | `maxReceiveCount: 5` too aggressive | Premature DLQ routing, lost processing | **HIGH** |\n| 2 | Source queue retention 4 days | Silent message expiry during outage | **HIGH** |\n| 3 | VisibilityTimeout may not cover processing time | Duplicate processing | **MEDIUM** |\n| 4 | `Principal: \"*\"` in queue policy | Latent public write exposure | **MEDIUM** |\n| 5 | No DLQ alarm, DLQ already has messages | Silent failure accumulation | **MEDIUM** |\n| 6 | 14 messages in source queue, no depth alarm | Undetected consumer stall | **LOW** |\n| 7 | DLQ short polling enabled | API waste, potential missed messages | **LOW** |",
"elapsed_s": 64.0
},
{
"queue": "08-clean-standard",
"condition": "treatment",
"trial": 1,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON, maxReceiveCount=5, and all string-typed second values including VisibilityTimeout=180, retention periods, etc.",
"Correctly reports no findings, matching the ground-truth clean result.",
"Does not flag the wildcard principal as a false positive; explicitly recognizes aws:SourceArn ArnEquals condition as a legitimate SNS-to-SQS narrowing pattern.",
"No false severity ranking issues; since there are no findings, no miscritical headline is possible and the agent correctly produces an empty findings list.",
"Names multiple boundary items matching ground-truth joins: consumer processing time vs visibility timeout, DLQ alerting/ownership, IAM graph, producer correctness, and CloudWatch metrics over time.",
"Frames consumer processing time, DLQ depth, and producer correctness as open questions requiring instrumentation rather than confirmed bugs.",
"Top recommendation is correctly 'none' \u2014 no fix invented for a clean queue, matching ground-truth guidance."
],
"agent_output": "# SQS Queue Audit: `notification-fanout`\n\n---\n\n## Step 1: Parse the Attributes\n\n**Source queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout` | standard queue |\n| `VisibilityTimeout` | `\"180\"` | 180 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `ReceiveMessageWaitTimeSeconds` | `\"20\"` | long-polling enabled |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `RedrivePolicy` | `{\"deadLetterTargetArn\":\"\u2026:notification-fanout-dlq\",\"maxReceiveCount\":5}` | DLQ wired; maxReceiveCount = 5 |\n| `Policy` | `{\"Statement\":[{\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:sns:\u2026:account-events\"}}}]}` | wildcard principal, narrowed by `ArnEquals` condition |\n\n**Dead-letter queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq` | standard queue |\n| `MessageRetentionPeriod` | `\"1209600\"` | 1,209,600 s (14 days) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `RedriveAllowPolicy` | `{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"\u2026:notification-fanout\"]}` | scoped correctly to source |\n\n---\n\n## Step 2: Audit the Redrive Path\n\n**R1 \u2014 No DLQ:** Not triggered. `RedrivePolicy` is present and points at a real queue.\n\n**R2 \u2014 maxReceiveCount out of band:** `maxReceiveCount = 5`. This sits within the 3\u201310 band. Not triggered.\n\n**R3 \u2014 DLQ retention not longer than source:**\n\n- Source retention: **345,600 s (4 days)**\n- DLQ retention: **1,209,600 s (14 days)**\n\nDLQ retention (14 d) > source retention (4 d). The ordering is correct; messages that fail late in the source window arrive in the DLQ with time to spare. Not triggered.\n\n---\n\n## Step 3: Audit the Message Lifecycle\n\n**R4 \u2014 Poison messages age out before reaching the DLQ:**\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout vs MessageRetentionPeriod\n5 \u00d7 180 s = 900 s (15 minutes) vs 345,600 s (4 days)\n```\n\n900 s \u226a 345,600 s. The product is far below retention; a poison message will exhaust all five receive attempts long before its age limit. Not triggered.\n\n**R5 \u2014 Visibility timeout at the 30 s default:** `VisibilityTimeout = 180 s`. Not at the 30 s default. Not triggered.\n\n**R6 \u2014 Retention shorter than a plausible outage:** Source retention is 4 days (345,600 s), well above the 1-hour threshold. Not triggered.\n\n---\n\n## Step 4: Audit Exposure\n\n**R7 \u2014 Open resource policy:**\n\nThe policy statement has `\"Principal\": \"*\"`, which would normally be a high-severity finding. However, it is narrowed by:\n\n```json\n\"Condition\": {\n \"ArnEquals\": {\n \"aws:SourceArn\": \"arn:aws:sns:eu-west-1:211125758836:account-events\"\n }\n}\n```\n\nThis is the standard SNS-to-SQS subscription pattern. `ArnEquals` on `aws:SourceArn` is a specific, effective narrowing condition; only the named SNS topic can use this permission. The methodology explicitly states this pattern must not be flagged. Not triggered.\n\n**R8 \u2014 Encryption at rest disabled:** `SqsManagedSseEnabled = \"true\"` on both the source queue and the DLQ. Not triggered.\n\n---\n\n## Step 5: Audit FIFO Invariants\n\n**R9 \u2014 FIFO with content-based dedup off:** Neither `FifoQueue` nor `ContentBasedDeduplication` attributes are present. This is a standard queue. Not triggered.\n\n---\n\n## Step 6: Rank, Report, and Name the Boundary\n\n---\n\n## Audit Report\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2014 `arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq` |\n\n---\n\n### Findings\n\n**No findings.**\n\nAll nine rules pass against this configuration:\n\n| Rule | Check | Result |\n|---|---|---|\n| R1 | DLQ present | \u2713 Pass |\n| R2 | `maxReceiveCount` = 5, within 3\u201310 | \u2713 Pass |\n| R3 | DLQ retention (14 d) > source retention (4 d) | \u2713 Pass |\n| R4 | 5 \u00d7 180 s = 900 s \u226a 345,600 s source retention | \u2713 Pass |\n| R5 | Visibility timeout = 180 s, not at 30 s default | \u2713 Pass |\n| R6 | Source retention = 4 days, above 1-hour floor | \u2713 Pass |\n| R7 | Wildcard principal narrowed by `aws:SourceArn` condition | \u2713 Pass |\n| R8 | SSE-SQS enabled on both source and DLQ | \u2713 Pass |\n| R9 | Standard queue; FIFO invariants not applicable | \u2713 Pass |\n\nThe redrive arithmetic is particularly clean: a poison message exhausts its five receive attempts in at most 15 minutes, then lands in a DLQ with 14 days of runway and at least 3 days and 45 minutes of that retention remaining (source retention minus the 15-minute exhaustion window). The retention ordering and the lifecycle product both have large safety margins.\n\n---\n\n### Boundary\n\nA clean configuration is not a clean system. The following questions are real and open; none of them are answerable from queue attributes alone.\n\n1. **Consumer processing time vs. visibility timeout.** The visibility timeout is 180 seconds. Whether that is sufficient depends entirely on how long the worker fleet takes to process a single notification. If any processing path routinely exceeds 180 seconds \u2014 due to downstream API latency, retry loops, or large payloads \u2014 the message will reappear and be delivered a second time. This is not visible in the queue config. It requires either timing instrumentation on the consumer side or CloudWatch metrics (`ApproximateAgeOfOldestMessage`, `NumberOfMessagesNotVisible`).\n\n2. **Live behaviour and queue health.** There are currently 14 messages in-flight and 1 message sitting in the DLQ. The static config cannot say whether the DLQ message is a known stale artifact, an active failure, or the leading edge of a processing regression. CloudWatch metrics (`NumberOfMessagesSent`, `NumberOfMessagesDeleted`, `ApproximateNumberOfMessagesNotVisible` over time) and an alert on DLQ depth are necessary to distinguish between them.\n\n3. **DLQ is unwatched.** The configuration confirms the DLQ is correctly wired and sized. It cannot confirm that anyone is alerted when messages arrive in it, or that any operational process exists to drain, inspect, and replay DLQ messages. A dead-letter queue that fills silently is the same as no dead-letter queue from an operational standpoint.\n\n4. **Effective IAM access is the union of this policy and every identity policy in the account.** The resource policy here grants `sqs:SendMessage` only to the named SNS topic. But any IAM principal with an identity policy that permits `sqs:SendMessage` or `sqs:ReceiveMessage` on this ARN can also act on the queue. That inventory is not visible here.\n\n5. **Producer correctness.** The queue receives from `arn:aws:sns:eu-west-1:211125758836:account-events`. Whether that topic is publishing the right events, at the right rate, with the right message attributes, and without duplicates is a property of the producer and the topic subscription filter policy \u2014 none of which are queue attributes.\n\n6. **Consumer fleet composition and failure modes.** Whether the worker fleet handles partial failures correctly (i.e., does not acknowledge a message until processing is genuinely complete), what happens to a message when a worker crashes mid-processing, and whether the fleet scales proportionally to queue depth are all behind this boundary.",
"elapsed_s": 49.6
},
{
"queue": "08-clean-standard",
"condition": "treatment",
"trial": 2,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON, maxReceiveCount, all string-typed second values, and DLQ retention.",
"Correctly reports no findings, matching the ground-truth clean result.",
"Correctly identifies the wildcard principal as narrowed by aws:SourceArn and does not flag it as a public exposure.",
"No silent-message-loss defects exist; agent correctly avoids inventing or mis-ranking any finding.",
"Names multiple things outside config scope: IAM graph, CloudWatch metrics, SNS subscription integrity, DLQ drainage \u2014 matching the ground-truth boundary.",
"Soft observations (long-poll on DLQ at 0, visibility timeout adequacy) are presented as trade-offs to verify, not confirmed bugs.",
"Top fix is correctly 'none' \u2014 no recommendation invented, matching the ground-truth clean control."
],
"agent_output": "# SQS Queue Audit: `notification-fanout`\n\n---\n\n## Step 1: Parse the Attributes\n\n**Source queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout` |\n| `VisibilityTimeout` | `\"180\"` | 180 s (3 min) |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `ReceiveMessageWaitTimeSeconds` | `\"20\"` | 20 s (long polling enabled) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `FifoQueue` | absent | Standard queue |\n| `RedrivePolicy` | JSON string | `deadLetterTargetArn`: `\u2026:notification-fanout-dlq`, `maxReceiveCount`: 5 |\n| `Policy` | JSON string | One statement \u2014 see below |\n\n**Resource policy statement (parsed):**\n\n```\nSid: AllowSnsTopic\nEffect: Allow\nPrincipal: \"*\"\nAction: sqs:SendMessage\nResource: arn:aws:sqs:eu-west-1:211125758836:notification-fanout\nCondition: ArnEquals \u203a aws:SourceArn = arn:aws:sns:eu-west-1:211125758836:account-events\n```\n\nPrincipal is `\"*\"` **with** a narrowing `aws:SourceArn` condition pinned to the SNS topic. This is the canonical SNS-to-SQS subscription pattern. R7 must **not** be fired.\n\n**Dead-letter queue**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | string | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq` |\n| `VisibilityTimeout` | `\"180\"` | 180 s |\n| `MessageRetentionPeriod` | `\"1209600\"` | 1,209,600 s (14 days) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `RedriveAllowPolicy` | JSON string | `byQueue`, restricted to source queue ARN \u2014 correctly scoped |\n\n---\n\n## Step 2: Apply the Rules\n\n### R1 \u2014 DLQ present?\n`RedrivePolicy` is present and points to `notification-fanout-dlq`. **Not fired.**\n\n### R2 \u2014 `maxReceiveCount` in band (3\u201310)?\n`maxReceiveCount` = 5. Within the 3\u201310 band. **Not fired.**\n\n### R3 \u2014 DLQ retention longer than source?\n- Source retention: 345,600 s (4 days)\n- DLQ retention: 1,209,600 s (14 days)\n- 1,209,600 > 345,600 \u2713\n\n**Not fired.**\n\n### R4 \u2014 Poison messages age out before reaching the DLQ?\nArithmetic: `maxReceiveCount \u00d7 VisibilityTimeout` vs `MessageRetentionPeriod`\n\n```\n5 \u00d7 180 = 900 s (15 min)\nMessageRetentionPeriod = 345,600 s (4 days)\n900 < 345,600 \u2713\n```\n\nA poison message consumes at most 900 seconds of wall-clock exhausting its receive count; the source queue retains for 4 days. Retention wins comfortably. **Not fired.**\n\n### R5 \u2014 Visibility timeout at the 30 s default?\n`VisibilityTimeout` = 180 s. Not the default. **Not fired.**\n\n### R6 \u2014 Retention shorter than a plausible outage?\n`MessageRetentionPeriod` = 345,600 s = 4 days. Well above 1 hour. **Not fired.**\n\n### R7 \u2014 Open resource policy (wildcard principal, no condition)?\nPrincipal is `\"*\"` but the `ArnEquals / aws:SourceArn` condition tightly scopes the grant to a single SNS topic ARN. This is explicitly excluded from R7 by the methodology. **Not fired.**\n\n### R8 \u2014 Encryption at rest disabled?\n`SqsManagedSseEnabled` = `\"true\"` on both source and DLQ. **Not fired.**\n\n### R9 \u2014 FIFO with content-based dedup off?\n`FifoQueue` attribute is absent; this is a standard queue. **Not fired.**\n\n---\n\n## Output\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout` |\n| **Type** | Standard (not FIFO) |\n| **DLQ wired** | Yes \u2192 `arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq` |\n| **Role** | SNS subscriber / worker-fleet source |\n\n---\n\n### Findings\n\n**No findings.**\n\nEvery rule was evaluated against the parsed configuration. All nine checks either passed their inequality or were inapplicable to this queue type. The specific judgments that matter most for this topology:\n\n- The R3 retention ordering is correct (DLQ retains 14 days, source retains 4 days), so late failures land in the DLQ with 10 days of headroom rather than expiring on arrival.\n- The R4 arithmetic is clear (worst-case exhaustion: 15 minutes against a 4-day retention budget), so a poison message cannot age out of the source queue before completing the redrive path.\n- The resource policy wildcard is correctly conditioned on `aws:SourceArn` and is the standard SNS subscription pattern, not an exposure.\n\nTwo configuration choices are worth noting \u2014 not as findings, but as deliberate trade-offs visible in the config:\n\n- **Long polling (`ReceiveMessageWaitTimeSeconds = 20`)** is correctly set on the source queue. The DLQ has it at 0, which is normal: DLQ consumers are typically manual or ad-hoc.\n- **`maxReceiveCount = 5`** is in the middle of the safe band. A single transient failure will not dead-letter a message, and a genuine poison message is quarantined after at most five attempts.\n\n---\n\n### Boundary\n\nThis audit read the static configuration of one queue and its dead-letter queue. The following questions are not answerable from those attributes alone, and a clean configuration is not a clean system:\n\n1. **Consumer processing time.** The 180 s visibility timeout was checked against the default (R5 \u2014 not the default, no flag). Whether 180 s is actually sufficient depends on how long the worker fleet takes to process a notification. If processing routinely exceeds 180 s, messages re-appear as duplicates mid-processing. This audit cannot observe that; CloudWatch's `ApproximateAgeOfOldestMessage` and consumer-side timing metrics can.\n\n2. **Live behaviour and queue depth.** The snapshot shows 14 messages in-flight on the source queue and 1 in the DLQ. Whether that DLQ message is stale, actively failing, or an artifact of a one-off error requires inspecting `SentTimestamp` and the CloudWatch `NumberOfMessagesSent` / `NumberOfMessagesDeleted` series \u2014 neither of which is a queue attribute.\n\n3. **Effective IAM.** The resource policy grants `sqs:SendMessage` to the SNS topic via `aws:SourceArn`. That is the write path. The read path \u2014 which IAM roles on the worker fleet have `sqs:ReceiveMessage`, `sqs:DeleteMessage`, and `sqs:ChangeMessageVisibility` \u2014 is determined by identity policies attached to those roles, which are not visible in this audit. The effective permission set is the union of the resource policy (visible here) and every identity policy in the account (not visible here).\n\n4. **DLQ operations.** The DLQ's `RedriveAllowPolicy` correctly restricts redrive permission to the source queue ARN. What this audit cannot confirm is whether any process or operator is actively consuming and actioning the DLQ. A DLQ that receives messages and is never drained is operationally equivalent to no DLQ. This is a people-and-process question, not a configuration question.\n\n5. **SNS subscription integrity.** The resource policy is correctly scoped to `arn:aws:sns:eu-west-1:211125758836:account-events`. Whether that SNS topic actually has a subscription pointing at this queue \u2014 and whether the subscription's `RawMessageDelivery` setting matches what the worker fleet expects \u2014 requires reading the SNS topic's attributes and subscription list, which are outside this queue's configuration.\n\n6. **Producer contract.** This is a standard queue, so deduplication is not enforced at the queue level. Whether duplicate notifications from SNS (e.g. during SNS retry on a failed delivery) are idempotent for the worker fleet is a property of the consumer implementation, not visible here.",
"elapsed_s": 51.7
},
{
"queue": "08-clean-standard",
"condition": "treatment",
"trial": 0,
"score": 7,
"scores_by_item": [
1,
1,
1,
1,
1,
1,
1
],
"notes": [
"Correctly parses RedrivePolicy JSON, maxReceiveCount=5, and all string-typed seconds including retention and visibility timeout.",
"Correctly reports zero findings, matching the ground truth that this is a clean queue.",
"Explicitly recognizes the wildcard principal with aws:SourceArn condition as the legitimate SNS-to-SQS pattern and does not flag it as a public exposure.",
"No false severity rankings introduced; the audit correctly treats this as a clean configuration with no defects to rank.",
"Names multiple boundary items including IAM graph access, consumer processing time vs visibility timeout, and CloudWatch metrics \u2014 matching the ground-truth boundary.",
"Observation about DLQ long-polling is correctly presented as a non-finding note, not a confirmed bug, demonstrating appropriate honesty about soft flags.",
"Top fix is correctly 'None' \u2014 the agent does not invent a recommendation for a clean queue, matching the ground-truth fix of 'None'."
],
"agent_output": "# SQS Queue Audit: notification-fanout\n\n---\n\n## Step 0 \u2014 Parse\n\nBefore judgment, extract and type every value.\n\n**Source queue**\n\n| Attribute | Raw string | Parsed |\n|---|---|---|\n| `VisibilityTimeout` | `\"180\"` | 180 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `SqsManagedSseEnabled` | `\"true\"` | enabled |\n| `KmsMasterKeyId` | absent | \u2014 |\n| `FifoQueue` | absent | standard queue |\n| `ContentBasedDeduplication` | absent | n/a |\n| `RedrivePolicy` | JSON string | `deadLetterTargetArn = arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq`, `maxReceiveCount = 5` |\n| `Policy` | JSON string | one statement, `Principal: \"*\"`, `Action: sqs:SendMessage`, `Condition: ArnEquals aws:SourceArn = arn:aws:sns:eu-west-1:211125758836:account-events` |\n\n**Dead-letter queue**\n\n| Attribute | Raw string | Parsed |\n|---|---|---|\n| `MessageRetentionPeriod` | `\"1209600\"` | 1,209,600 s (14 days) |\n| `VisibilityTimeout` | `\"180\"` | 180 s |\n| `SqsManagedSseEnabled` | `\"true\"` | enabled |\n| `RedriveAllowPolicy` | JSON string | `byQueue`, source = notification-fanout ARN |\n\n---\n\n## Step 1 \u2014 Redrive path\n\n**R1 \u2014 No DLQ?** `RedrivePolicy` is present and correctly references the DLQ. \u2713 Not fired.\n\n**R3 \u2014 DLQ retention not longer than source retention?**\n- Source: 345,600 s (4 days)\n- DLQ: 1,209,600 s (14 days)\n- 1,209,600 > 345,600 \u2713 Not fired.\n\n**R2 \u2014 `maxReceiveCount` outside 3\u201310?**\n- `maxReceiveCount = 5`. Within the 3\u201310 band. \u2713 Not fired.\n\n---\n\n## Step 2 \u2014 Message lifecycle\n\n**R4 \u2014 Poison messages age out before reaching the DLQ?**\n\nArithmetic: `maxReceiveCount \u00d7 VisibilityTimeout` vs `MessageRetentionPeriod`\n\n```\n5 \u00d7 180 = 900 s\nMessageRetentionPeriod = 345,600 s\n900 < 345,600\n```\n\n900 s is well under 345,600 s. \u2713 Not fired.\n\n**R5 \u2014 Visibility timeout at the 30 s default?**\n- `VisibilityTimeout = 180 s`. Not the default. \u2713 Not fired.\n\n**R6 \u2014 Retention shorter than a plausible outage?**\n- `MessageRetentionPeriod = 345,600 s = 4 days`. Well above 1 hour. \u2713 Not fired.\n\n---\n\n## Step 3 \u2014 Exposure\n\n**R7 \u2014 Open resource policy (wildcard principal, no condition)?**\n\nThe single policy statement reads:\n\n```json\n{\n \"Effect\": \"Allow\",\n \"Principal\": \"*\",\n \"Action\": \"sqs:SendMessage\",\n \"Condition\": {\n \"ArnEquals\": {\n \"aws:SourceArn\": \"arn:aws:sns:eu-west-1:211125758836:account-events\"\n }\n }\n}\n```\n\n`Principal` is `\"*\"` but there is a narrowing `Condition` \u2014 specifically `aws:SourceArn` scoped to a single SNS topic ARN in the same account. This is the standard, correct SNS-to-SQS subscription pattern. The methodology explicitly states this must **not** be flagged. \u2713 Not fired.\n\n**R8 \u2014 Encryption at rest disabled?**\n- `SqsManagedSseEnabled = true` on both source and DLQ. \u2713 Not fired.\n\n---\n\n## Step 4 \u2014 FIFO invariants\n\n`FifoQueue` is absent \u2014 this is a standard queue. R9 is not applicable. \u2713 Not fired.\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:notification-fanout` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2014 `arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq` |\n| **Role** | SNS subscriber (`account-events` topic) \u2192 worker fleet |\n\n---\n\n### Findings\n\n**No findings.**\n\nEvery check across all nine rules passes cleanly against the supplied configuration:\n\n| Rule | Check | Result |\n|---|---|---|\n| R1 | DLQ present | \u2713 Pass |\n| R2 | `maxReceiveCount = 5` within 3\u201310 | \u2713 Pass |\n| R3 | DLQ retention (14 d) > source retention (4 d) | \u2713 Pass |\n| R4 | `5 \u00d7 180 s = 900 s` < `345,600 s` retention | \u2713 Pass |\n| R5 | Visibility timeout 180 s, not the 30 s default | \u2713 Pass |\n| R6 | Retention 4 days, above the 1-hour floor | \u2713 Pass |\n| R7 | Wildcard principal is condition-narrowed to a single SNS ARN | \u2713 Pass |\n| R8 | SQS-managed SSE enabled on source and DLQ | \u2713 Pass |\n| R9 | Standard queue \u2014 not applicable | \u2713 n/a |\n\nOne observation that does not rise to a finding, but is worth recording:\n\n> **DLQ long-polling is off.** The DLQ's `ReceiveMessageWaitTimeSeconds = 0`, while the source queue uses `20` (long-polling). This has no effect on message safety \u2014 the DLQ is a quarantine store, not a normal consumer path \u2014 but if you run automated DLQ monitoring or alerting via a poller, configuring long-polling there too will reduce empty-receive API charges and CPU spin. Not a rule violation; noted for completeness.\n\n---\n\n### Boundary\n\nA clean configuration is not a clean system. The following questions are unanswerable from the queue attributes alone, and each is a live risk vector that the static audit cannot close:\n\n1. **Consumer processing time vs. visibility timeout.** The 180 s visibility timeout looks deliberate rather than defaulted, but whether it actually covers the worker fleet's processing time is a property of the consumers, not the queue. If any worker takes longer than 180 s, messages reappear mid-processing and are delivered twice. Confirm by measuring the 99th-percentile processing latency of the worker fleet and comparing it to 180 s. This join requires CloudWatch or application instrumentation.\n\n2. **Live redrive volume and DLQ drain discipline.** The DLQ currently holds 1 message. Whether that message is being monitored, alarmed on, and drained by an owner is invisible to this audit. A DLQ that fills silently and is never consumed defeats the quarantine entirely. The join requires a CloudWatch alarm on `ApproximateNumberOfMessages` for the DLQ (with a non-zero threshold) and operational confirmation that someone owns the drain process.\n\n3. **Age of the oldest message.** The source queue reports 14 messages in flight. The `ApproximateAgeOfOldestMessage` metric would show whether any are approaching stall. That is a CloudWatch time-series, not an attribute, and is behind the boundary.\n\n4. **The full access-control picture.** The resource policy is correctly scoped to the SNS topic ARN. But the effective set of principals that can read from or manage this queue is the union of this resource policy and every identity policy in account `211125758836`. IAM role and user policies are not queue attributes. The audit cannot confirm that only the intended worker fleet has `sqs:ReceiveMessage`, or that no overly-permissive role grants `sqs:*` on `*`. A full access audit requires reading the account's IAM graph.\n\n5. **SNS subscription hygiene (producer side).** The resource policy authorises sends from the `account-events` SNS topic. Whether that topic's subscription to this queue is correctly configured (raw message delivery, filter policy, subscription confirmation) is a property of the SNS topic, not the SQS queue. If the subscription is unconfirmed or misconfigured, messages are dropped before they arrive \u2014 and the queue config cannot show that.\n\n6. **DLQ ownership and runbook.** `maxReceiveCount = 5` means a message that fails five times is quarantined in the DLQ and stays there for up to 14 days. Whether there is an operational runbook for inspecting, replaying, or discarding those messages \u2014 and who owns it \u2014 is not encoded anywhere in the queue attributes. A correctly-wired DLQ that nobody drains is operationally equivalent to silent loss on a longer time horizon.",
"elapsed_s": 55.5
}
]Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |