heymegabyte/agent-skills / skills/tdd-and-verification
tdd-and-verification
VerifiedCanonical Test-Driven Development (Red-Green-Refactor), legacy characterization-first retrofit, Nx selective impact execution, 300-second CI gate, Cloudflare Browser Run parallel CDP orchestration, and 15-family (450-item) verification policy.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.7 KB |
| references/f01-unit-and-business-behavior.md | markdown | 9.0 KB |
| references/f02-components-and-interactions.md | markdown | 8.8 KB |
| references/f03-integration-and-contracts.md | markdown | 8.1 KB |
| references/f04-system-and-e2e-journeys.md | markdown | 7.7 KB |
| references/f05-visual-regression-and-compatibility.md | markdown | 8.5 KB |
| references/f06-accessibility-and-usability.md | markdown | 7.6 KB |
| references/f07-generated-and-adversarial-inputs.md | markdown | 8.6 KB |
| references/f08-security-and-privacy.md | markdown | 8.8 KB |
| references/f09-data-correctness-and-distributed-behavior.md | markdown | 8.5 KB |
| references/f10-performance-capacity-and-cost.md | markdown | 8.1 KB |
| references/f11-resilience-failure-handling-and-recovery.md | markdown | 8.3 KB |
| references/f12-deployment-platform-packaging-and-infrastructure.md | markdown | 8.3 KB |
| references/f13-ai-agent-and-evaluation-behavior.md | markdown | 8.6 KB |
| references/f14-content-analytics-and-observability.md | markdown | 7.9 KB |
| references/f15-test-suite-effectiveness-and-skill-quality.md | markdown | 9.5 KB |
references/f15-test-suite-effectiveness-and-skill-quality.md
View on GitHub---
name: "F15 — Test suite effectiveness and testing skill quality"
familyId: "F15"
defaultLayer: "meta_harness"
defaultTier: "required_ci"
itemCount: 30
---
# F15: Test suite effectiveness and testing skill quality (F15.01–F15.30)
- **Default Verification Layer**: `meta_harness`
- **Default Execution Tier**: `required_ci`
- **Authorized Pilot Implementation**: `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts`
## Complete 30-Recommendation Obligation Matrix
| ID | Title | Obligation | Layer | Tier | Pilot Status | Evidence / Location |
|---|---|---|---|---|---|---|
| `F15.01` | **Red-Green-Refactor regression proof** | Prove new regression tests fail for the intended missing behavior before fix and pass after. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.02` | **Independent test oracle derivation** | Derive test expectations from specifications rather than copying implementation logic. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.03` | **Zero-test discovery failure gate** | Fail the CI gate immediately if any selected target or shard discovers 0 tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.04` | **Distinct skipped/blocked reporting** | Report skipped, cached, and environment-blocked checks separately from fresh passes. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.05` | **First-failure preservation on retry** | Preserve first failure evidence and classify retry passes as flaky rather than clean. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.06` | **Quarantine owner, expiry, and replacement** | Enforce QuarantineRecordSchema requiring owner, reason, expiresAtIso, and replacementTestId. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.07` | **Deterministic seed replay** | Record and replay random seeds for property and sampling tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.08` | **Cross-worker and shard state isolation** | Verify parallel Vitest and Playwright workers share zero mutable state. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.09` | **Mock and fixture contract verification** | Validate DataForSEO and theme fixtures against strict production Zod schemas. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.10` | **Cache input completeness and live exclusion** | Include lockfile/config/env in cache keys and never cache mutable live E2E browser runs. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.11` | **Conservative impact selection** | Combine dependency graph, explicit route/component maps, and critical sentinels in nx-selector. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.12` | **Test-only change execution** | Trigger affected test suites when spec files, fixtures, or test helpers change. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.13` | **Stale impact map invalidation** | Expand selection to full suite when impact mappings or source maps are stale. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.14` | **Verified git baseline resolution** | Validate baseline commit exists in git history; expand to full suite if missing. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.15` | **Duration-balanced shard partitioning** | Partition selected specs across shards with zero dropped or duplicated tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.16` | **Targeted mutation testing on critical logic** | Verify critical RBAC, word-count, and release-gate mutants are killed by tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.17` | **Requirement-to-assertion traceability** | Map all 450 F01–F15 items and 360 A01–L30 items to concrete code and tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.18` | **Positive and negative harness controls** | Prove the gate passes on valid bundles and fails on seeded defective counterexamples. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.19` | **Seeded defect skill benchmarking** | Benchmark the testing skill against seeded defects in skill-eval-harness.ts. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.20` | **Held-out oracle evaluation** | Evaluate agent test submissions against independent held-out assertions. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.21` | **Anti-weakening and anti-skip detection** | Detect weakened assertions, deleted tests, .skip/.only, and fabricated reports automatically. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.22` | **False-pass and false-alarm measurement** | Track precision, recall, runtime, and cost across all 15 skill evaluation scenarios. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.23` | **Commit, shard, and attempt artifact binding** | Bind commit SHA, shard index, and attempt number into GateReconciliationReport. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.24` | **Reviewed snapshot and baseline updates** | Require explicit behavioral justification for any snapshot or visual baseline change. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.25` | **Fail-closed on timeout or cancellation** | Treat timed-out, cancelled, or missing shards as blocking deployment failures. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.26` | **Built-code coverage verification** | Verify tests exercise actual exported package modules and built Worker/Storybook bundles. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.27` | **Decision-boundary condition coverage** | Test boundary conditions across RBAC roles, word counts, SERP overlap, and budgets. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.28` | **Independent acceptance review** | Challenge implementation + test pairs with independent adversarial counterexamples. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.29` | **Protected-behavior deduplication** | Consolidate overlapping assertions into high-signal tests without losing failure localization. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.30` | **Periodic full-audit selector calibration** | Compare selective execution against full workspace runs to verify zero missed regressions. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |