Skip to content

Destynova2/cli-code-skills / cli-audit-test

cli-audit-test

Verified

Audit test plan quality and maturity. Scores coverage, techniques, pyramid balance, negative testing, NFR, automation, CI integration. Use when reviewing a test plan, test strategy, test suite structure, or saying 'audit tests', 'test quality', 'test plan review', 'test maturity', 'are my tests good enough', 'test scoring', 'test pyramid'. Also triggers on 'test coverage', 'test gaps', 'missing tests'.

FileKindSize
SKILL.mdskill md10.3 KB
references/anti-patterns.mdmarkdown7.9 KB
references/dimensions.mdmarkdown14.1 KB
references/scoring-framework.mdmarkdown8.9 KB
references/techniques.mdmarkdown8.9 KB

references/anti-patterns.md

View on GitHub
# Anti-Pattern Detection Rules

> **When to read:** During Step 4 when scanning for anti-patterns after scoring dimensions.

---

## Contents

- Test isolation — the chaperone principle
- 11 Anti-Patterns
- Detection Heuristics
- Recommendation Templates

---

## Test isolation — the chaperone principle

> **Biomimetic frame — molecular chaperones (HSP60/HSP70).** When a protein misfolds in the cell, chaperones do not patch the broken structure in place — patching is impossible because misfolding is structural, not local. Instead, they fully unfold the protein and let it re-fold from a clean state. Sometimes this happens many times before the protein folds correctly; if it never does, the cell ubiquitinates it for destruction. The principle: **patching corrupted state never works, only a full reset does.**
>
> Tests are the same. A test that depends on residual state from a previous test is a half-folded protein looking for a chaperone that does not exist. The only sustainable strategy is for every test to start from a fully clean state — fresh database, fresh fixtures, fresh in-memory caches, fresh environment. Anything else is patching.

This frame underlies two of the anti-patterns below: **State Bleed** (the structural cause) and **Flaky Acceptance** (the visible symptom). Test Data Leakage is the third leg, focused on the security side of the same hygiene problem.

---

## 11 Anti-Patterns

Flag these known anti-patterns with severity and evidence:

| Anti-Pattern | Source | Detection | Severity |
|--------------|--------|-----------|----------|
| Ice Cream Cone | Codepipes | Count tests per level, check ratio — more E2E than unit = flagged | Critical |
| Cassandre Syndrome | La Taverne | Tests exist but results are never analyzed (no reporting/gating in CI) | Critical |
| Flaky Acceptance | Codepipes #6 | Retry logic in tests, `@retry`, `flaky` markers, `sleep` in test code | Warning |
| Testing Implementation | Codepipes #4 | Tests break on refactor (coupled to internals, mock overuse) | Warning |
| 100% Coverage Obsession | Codepipes #5 | Coverage targets without quality checks on what's covered | Warning |
| Late Testing | ISTQB shift-left | Tests only at E2E level, no early feedback, no unit tests | Critical |
| Smoke-less Builds | La Taverne | No smoke test suite, CI runs full suite on every commit | Warning |
| Combinatorial Explosion | La Taverne (Pairwise) | N parameters x M values tested exhaustively instead of pairwise | Info |
| Test Data Leakage | TMMi | Hardcoded secrets, shared mutable state between tests | Warning |
| State Bleed | Chaperone principle | Test outcome depends on residual state from a previous test — order dependency, missing teardown, shared in-memory singletons, persistent fixtures across tests. Structural cause behind most flaky tests | Critical |
| Silent Drift Blindness | D13 (Biological) | No mutation/contract/property testing — only example-based tests, silent behavioral changes go undetected | Critical |

## Detection Heuristics

### Ice Cream Cone
- Count test files in `unit/`, `integration/`, `e2e/` directories
- If E2E count > unit count, flag as Ice Cream Cone
- Also check for manual test plans without corresponding automated tests

### Cassandre Syndrome
- Check CI config for test result publishing
- Look for quality gates (PR blocked on failure)
- If tests run but no one acts on results → Cassandre

### Flaky Acceptance
- Search for `@retry`, `@flaky`, `retry:`, `retries:` in test files
- Search for `sleep`, `wait`, `setTimeout` in test code (not helpers)
- Check CI logs for intermittent failures if accessible

### Testing Implementation (Coupling)
- High mock count relative to test count
- Tests that reference private/internal APIs
- Tests that break when code is refactored without behavior change

### 100% Coverage Obsession
- Coverage threshold set to 100% in CI config
- Trivial tests (testing getters/setters, testing framework code)
- Coverage metric without mutation testing or assertion quality checks

### Late Testing
- No unit test framework in dependencies
- All tests are E2E or manual
- Tests added only after bugs are found (reactive, not proactive)

### Smoke-less Builds
- No `smoke` tag/marker/directory in test suite
- CI runs full test suite on every push (no fast feedback loop)
- No tiered test execution strategy

### Combinatorial Explosion
- Large test matrices without pairwise reduction
- Exhaustive parameter combinations in test names
- No PICT or similar combinatorial tool in project

### Test Data Leakage
- Hardcoded passwords, tokens, or API keys in test files
- Tests sharing global state (database, files) without cleanup
- Test order dependencies (test B fails if test A doesn't run first)

### State Bleed
- Test files that must run in a specific order — look for `@TestMethodOrder`, `@FixMethodOrder`, `pytest.mark.run(order=N)`, manual ordering in `tests/main.rs`
- Tests that pass in isolation but fail in suite (or vice versa) — check CI logs for `FAILED ... PASSED on retry` patterns
- Module-level fixtures or `static`/`lazy_static`/`OnceCell` mutable state in test files
- Missing teardown: setup functions without matching teardown, `setUp` without `tearDown`, fixtures with no `finalizer`/`yield ... cleanup`
- Database tests without per-test transaction rollback or per-test schema reset
- Tests that mutate environment variables, global config, or `chdir` without restoring on exit
- Filesystem tests that write to `/tmp` with hardcoded names instead of unique-per-test directories
- In-memory caches (Redis client, HTTP client pool, ORM session) reused across tests without reset
- **The "patch the state" smell:** any test code that does `if previous_run_left_X { clean_X() }` instead of always starting clean

### Silent Drift Blindness
- No `cargo-mutants` / `mutmut` / `Stryker` / `pitest` in dependencies or CI
- No `Pact` / `insta` / snapshot testing / consumer-driven contracts
- No `proptest` / `quickcheck` / `Hypothesis` / `fast-check` in dependencies
- All tests are example-based (specific inputs → specific expected outputs)
- No `.snap` files, no `pacts/` directory, no mutation score reports
- A function's return value can change and no test catches it unless the exact example was covered

## Recommendation Templates

| Anti-Pattern | Recommendation |
|--------------|---------------|
| Ice Cream Cone | Add unit tests for core logic. Target ratio: 70% unit, 20% integration, 10% E2E |
| Cassandre Syndrome | Add quality gates in CI. Block PRs on test failure. Publish test reports |
| Flaky Acceptance | Quarantine flaky tests. Fix root cause (usually timing). Use deterministic waits |
| Testing Implementation | Test behavior, not implementation. Reduce mocks. Use contract tests |
| 100% Coverage Obsession | Add mutation testing. Focus coverage on critical paths, not vanity metrics |
| Late Testing | Shift left: add unit tests during development. TDD for new features |
| Smoke-less Builds | Define a smoke suite (critical paths only). Run smoke on every commit, full suite on PR |
| Combinatorial Explosion | Adopt pairwise testing (PICT tool). Reduces test count by 80%+ with same defect detection |
| Test Data Leakage | Use test fixtures with cleanup. Generate data per test. Never hardcode secrets |
| State Bleed | Every test must start from a fully reset state — fresh DB transaction (rolled back at the end), fresh tempdir, fresh in-memory caches, restored env vars, no static singletons. Run the suite in a randomized order (`pytest -p randomly`, `cargo test -- --shuffle`, `--testRandomOrderSeed`) to surface hidden order dependencies. The chaperone rule: if a test cannot start from a clean state, it is not a test, it is a fragment of a previous test |
| Silent Drift Blindness | Add at least one drift detection layer: mutation testing (`cargo-mutants`) on critical paths, snapshot/contract testing (`insta`, `Pact`) on APIs, or property-based invariants (`proptest`) on core domain logic. Start with the layer that matches your biggest risk |
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 468kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 468kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 468kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall