anyshift-io/sre-skills / skills/sqs-queue-auditor
sqs-queue-auditor
Verifiedtests/eval/judge_prompt.md
View on GitHub# Judge prompt template
The runner fills this template per graded output. Reproduced here so a different judge implementation can be swapped in.
```
QUEUE: {id}
NAME: {queue}
KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): {expected_codes} (severity: {top_severity}; clean: {clean})
- What the defect is: {expected_headline}
- Correct top fix: {expected_top_fix}
- Boundary the audit cannot cross: {expected_boundary_join}
AGENT AUDIT OUTPUT:
{agent_output}
RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: parses the embedded RedrivePolicy and string-typed seconds correctly.
2. Findings: identifies the ground-truth defect(s) by substance (or correctly reports none for the clean control).
3. No false positives: does not assert a material misconfiguration the ground truth lacks (esp. not calling an aws:SourceArn-scoped wildcard "public").
4. Criticality: ranks silent-message-loss defects as most serious; does not headline a soft flag.
5. Boundary: names at least one thing it cannot determine from config alone, matching the ground-truth join.
6. Honesty on soft flags: presents flags that depend on consumers/data/producers as flags to verify, not confirmed bugs.
7. Recommendation: top fix matches the ground-truth fix in substance.
Return JSON only, no prose:
{
"scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
"notes": ["<one sentence per item>", ...]
}
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |