JPeetz/agent-skills / skills/code-quality/clean-pro-review-gates
clean-pro-review-gates
FlaggedUse this skill when reviewing any AI-generated or modified code, tests, docs, or infrastructure before merging. Applies a 5-gate review framework — code quality (15 LLM failure modes), tests, docs, security (OWASP Top 10:2025/CWE Top 25/LLM Top 10), and CI/CD/container/IaC hardening — with each gate mapping violations to published research on how LLMs systematically fail.
| File | Kind | Size |
|---|---|---|
| CHANGELOG.md | markdown | 1.9 KB |
| LICENSE | other | 1.1 KB |
| SKILL.md | skill md | 12.9 KB |
| evals/evals.json | config | 10.4 KB |
| references/code-quality-gate.md | markdown | 5.9 KB |
| references/docs-review-gate.md | markdown | 3.1 KB |
| references/infra-hardening-gate.md | markdown | 5.0 KB |
| references/security-review-gate.md | markdown | 6.8 KB |
| references/security-scan-notes.md | markdown | 1.1 KB |
| references/test-review-gate.md | markdown | 4.3 KB |
evals/evals.json
View on GitHub{
"version": "1.0.0",
"description": "Eval test cases for the clean-pro-review-gates combined meta-skill",
"evals": [
{
"name": "gate-1-code-quality-swallowed-errors",
"description": "Gate 1 should flag broad catch-all error handling that swallows exceptions",
"test_input": "def get_email(user_id):\n try:\n user = user_store.get(user_id)\n return user.email\n except Exception:\n return None",
"expected_output": "Rule 15 violation: broad catch-all that swallows all exceptions. Catch only the specific error type you can recover from.",
"assertions": [
"Detects broad except clause (except Exception, except Throwable, catch all)",
"Flags silent return of None/null as error swallowing",
"Suggests catching only the specific error type",
"Mentions Karpathy or 'swallowed errors' in reasoning"
]
},
{
"name": "gate-1-code-quality-hallucinated-import",
"description": "Gate 1 should flag imports that may not exist in the installed version",
"test_input": "from langchain_community.vectorstores import Qdrant\nfrom huggingface_cli import upload",
"expected_output": "Rule 17 / Rule 6 violation: verify every import against the installed version. AI-suggested package names may not exist or may be slopsquatting vectors.",
"assertions": [
"Flags unknown or AI-suggested package imports",
"Suggests verifying against lockfile or installed packages",
"Mentions package hallucination research (USENIX Security '25)"
]
},
{
"name": "gate-1-code-quality-mock-fallback",
"description": "Gate 1 should flag hardcoded success returns in production code",
"test_input": "def get_user_balance(user_id):\n return 1000 # TODO: actual provider call",
"expected_output": "Rule 18 violation: hardcoded 'success' return or fixture data in production code. Never return mock data from a function whose spec says it should perform real work.",
"assertions": [
"Flags hardcoded return value as mock fallback",
"Flags TODO as cover for incomplete implementation",
"Suggests raising NotImplementedError or explicit unimplemented mechanism"
]
},
{
"name": "gate-1-code-quality-paint-by-numbers-comments",
"description": "Gate 1 should flag comments that restate the code instead of explaining why",
"test_input": "# Increment counter by one\ncounter += 1\n\n# Step 3: return the result\nreturn result",
"expected_output": "Rule 5 violation: comments explain *why*, never *what*. Delete comments paraphrasing the line below them. Remove step-number scaffolding comments.",
"assertions": [
"Flags 'what-comments' that restate obvious code",
"Flags step-number scaffolding comments",
"Suggests deleting or converting to 'why' commentary"
]
},
{
"name": "gate-1-code-quality-naming-violation",
"description": "Gate 1 should flag generic, intent-less names",
"test_input": "def process_data(data):\n result = []\n for item in data:\n temp = item.value * 2\n result.append(temp)\n return result",
"expected_output": "Rule 1 violation: names must reveal intent. 'data', 'process_data', 'result', 'item', 'temp' are generic. Use domain-revealing names like 'double_prices(orders)'.",
"assertions": [
"Flags 'data', 'result', 'item', 'temp' as generic names",
"Flags 'process_data' as verb without revealed intent",
"Suggests domain-specific replacement names"
]
},
{
"name": "gate-2-test-mock-of-internal-helper",
"description": "Gate 2 should flag mocks of internal helpers instead of system boundaries",
"test_input": "def test_calculate_total():\n with patch('app.services.calculator._apply_tax') as mock_tax:\n mock_tax.return_value = 10.0\n result = calculator.calculate_total(100)\n mock_tax.assert_called_once_with(100)",
"expected_output": "Rule 1/Rule 2 violation: testing implementation details by mocking an internal helper. Assert the return value or observable side effect from the caller's perspective instead.",
"assertions": [
"Flags mock of internal/private function as implementation test",
"Suggests asserting return value instead of mock call count",
"Notes that internal helper _apply_tax is not a system boundary"
]
},
{
"name": "gate-2-test-data-driven-opportunity",
"description": "Gate 2 should flag near-duplicate tests that should be merged into data-driven",
"test_input": "def test_discount_10():\n result = apply_discount(100, 10)\n assert result == 90\n\ndef test_discount_20():\n result = apply_discount(100, 20)\n assert result == 80\n\ndef test_discount_50():\n result = apply_discount(100, 50)\n assert result == 50",
"expected_output": "Rule 3 violation: three tests share identical setup and differ only in input/output values. Merge into a single data-driven test using parametrize/data provider.",
"assertions": [
"Detects near-identical test bodies",
"Suggests data-driven/parameterized test pattern",
"Notes shared setup and assertion pattern as merge signal"
]
},
{
"name": "gate-2-test-unjustified-constructor-test",
"description": "Gate 2 should flag tests that verify trivially obvious behavior",
"test_input": "def test_user_constructor():\n user = User(name=\"Alice\", email=\"alice@example.com\")\n assert user.name == \"Alice\"\n assert user.email == \"alice@example.com\"",
"expected_output": "Rule 4 violation: this test catches no bug that wouldn't be caught by the type system or a single integration test. Test behavior, not attribute assignment.",
"assertions": [
"Flags constructor/attribute tests as unjustified",
"Notes that the type system or a single integration test covers this",
"Suggests deleting the test or replacing with a behavior test"
]
},
{
"name": "gate-3-docs-symbol-verification",
"description": "Gate 3 should flag docs that reference symbols not in the source code",
"test_input": "The `POST /api/v2/invoices` endpoint returns a list of invoices. Use the `generate_report_v2()` helper to create PDFs.",
"expected_output": "Rule 1 violation: verify every referenced symbol against the actual source. If `POST /api/v2/invoices` or `generate_report_v2()` do not exist in the code, the docs contain false claims.",
"assertions": [
"Flags unverifiable endpoint references",
"Flags unverifiable function references",
"Requires verification against actual source, not memory"
]
},
{
"name": "gate-3-docs-code-sample-runnable",
"description": "Gate 3 should flag code samples with imports that don't exist or hardcoded paths",
"test_input": "```python\nfrom nonexistent_library import process\nresult = process('/Users/alice/local/data.csv')\nprint(result)\n```",
"expected_output": "Rule 2 violation: code sample must work on a clean machine. Import 'nonexistent_library' must be checked. Hardcoded local paths ('/Users/alice/...') must not ship.",
"assertions": [
"Flags unverifiable imports in code samples",
"Flags hardcoded local paths",
"Requires sample to be runnable outside author's machine"
]
},
{
"name": "gate-4-security-sql-injection",
"description": "Gate 4 should flag SQL injection via string concatenation",
"test_input": "query = \"SELECT * FROM users WHERE email = '\" + email + \"'\"\ndb.execute(query)",
"expected_output": "Rule 2 violation: string-built query with input is SQL injection. Use parameterized queries/prepared statements.",
"assertions": [
"Flags string concatenation in SQL query",
"Suggests parameterized query",
"Maps to CWE-89"
]
},
{
"name": "gate-4-security-hardcoded-secret",
"description": "Gate 4 should flag hardcoded secrets in source code",
"test_input": "API_KEY = \"sk_live_51Hreal_key_here\"\nclient = Client(api_key=\"AKIArealsecret\")",
"expected_output": "Rule 11 violation: secret as literal in source code. Read from environment or secrets manager. Any secret that touched a commit is compromised — flag for rotation.",
"assertions": [
"Flags hardcoded API keys and tokens",
"Suggests environment variable or secrets manager",
"Maps to CWE-798"
]
},
{
"name": "gate-4-security-missing-authorization",
"description": "Gate 4 should flag endpoints without server-side authorization checks",
"test_input": "@app.get('/api/invoices/{id}')\nasync def get_invoice(id: int):\n invoice = await db.get(id)\n return invoice",
"expected_output": "Rule 7 violation: missing authorization check. Any logged-in user can access any invoice (IDOR). Verify the current principal owns or may act on the resource.",
"assertions": [
"Flags missing ownership/permission check",
"Identifies IDOR vulnerability potential",
"Maps to CWE-862, CWE-639"
]
},
{
"name": "gate-5-infra-script-injection",
"description": "Gate 5 should flag untrusted context in shell inline",
"test_input": "jobs:\n comment:\n runs-on: ubuntu-latest\n steps:\n - run: \"echo \\\"Issue: ${{ github.event.issue.title }}\\\"\"",
"expected_output": "Rule 1 violation: attacker-controlled workflow context interpolated directly into shell. Issue title is attacker-controllable; pass through env var and quote.",
"assertions": [
"Flags ${{ }} interpolation in run: step",
"Identifies attacker-controlled context",
"Suggests env var passthrough as mitigation",
"Maps to CICD-SEC-4"
]
},
{
"name": "gate-5-infra-unpinned-action",
"description": "Gate 5 should flag third-party actions pinned to a moving tag",
"test_input": "steps:\n - uses: actions/checkout@v4\n - uses: docker/login-action@v2\n - uses: actions/setup-node@main",
"expected_output": "Rule 4 violation: third-party actions pinned to a moving tag or branch (@v4, @v2, @main). Pin to a full commit SHA to prevent supply-chain hijack.",
"assertions": [
"Flags moving tag pinning",
"Flags branch pinning ('@main')",
"Suggests SHA pinning with version comment",
"Maps to CICD-SEC-3"
]
}
]
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |