Skip to content

heymegabyte/agent-skills / skills/tdd-and-verification

tdd-and-verification

Verified

Canonical Test-Driven Development (Red-Green-Refactor), legacy characterization-first retrofit, Nx selective impact execution, 300-second CI gate, Cloudflare Browser Run parallel CDP orchestration, and 15-family (450-item) verification policy.

references/f15-test-suite-effectiveness-and-skill-quality.md

View on GitHub
---
name: "F15 — Test suite effectiveness and testing skill quality"
familyId: "F15"
defaultLayer: "meta_harness"
defaultTier: "required_ci"
itemCount: 30
---

# F15: Test suite effectiveness and testing skill quality (F15.01–F15.30)

- **Default Verification Layer**: `meta_harness`
- **Default Execution Tier**: `required_ci`
- **Authorized Pilot Implementation**: `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts`

## Complete 30-Recommendation Obligation Matrix

| ID | Title | Obligation | Layer | Tier | Pilot Status | Evidence / Location |
|---|---|---|---|---|---|---|
| `F15.01` | **Red-Green-Refactor regression proof** | Prove new regression tests fail for the intended missing behavior before fix and pass after. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.02` | **Independent test oracle derivation** | Derive test expectations from specifications rather than copying implementation logic. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.03` | **Zero-test discovery failure gate** | Fail the CI gate immediately if any selected target or shard discovers 0 tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.04` | **Distinct skipped/blocked reporting** | Report skipped, cached, and environment-blocked checks separately from fresh passes. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.05` | **First-failure preservation on retry** | Preserve first failure evidence and classify retry passes as flaky rather than clean. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.06` | **Quarantine owner, expiry, and replacement** | Enforce QuarantineRecordSchema requiring owner, reason, expiresAtIso, and replacementTestId. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.07` | **Deterministic seed replay** | Record and replay random seeds for property and sampling tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.08` | **Cross-worker and shard state isolation** | Verify parallel Vitest and Playwright workers share zero mutable state. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.09` | **Mock and fixture contract verification** | Validate DataForSEO and theme fixtures against strict production Zod schemas. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.10` | **Cache input completeness and live exclusion** | Include lockfile/config/env in cache keys and never cache mutable live E2E browser runs. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.11` | **Conservative impact selection** | Combine dependency graph, explicit route/component maps, and critical sentinels in nx-selector. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.12` | **Test-only change execution** | Trigger affected test suites when spec files, fixtures, or test helpers change. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.13` | **Stale impact map invalidation** | Expand selection to full suite when impact mappings or source maps are stale. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.14` | **Verified git baseline resolution** | Validate baseline commit exists in git history; expand to full suite if missing. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.15` | **Duration-balanced shard partitioning** | Partition selected specs across shards with zero dropped or duplicated tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.16` | **Targeted mutation testing on critical logic** | Verify critical RBAC, word-count, and release-gate mutants are killed by tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.17` | **Requirement-to-assertion traceability** | Map all 450 F01–F15 items and 360 A01–L30 items to concrete code and tests. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.18` | **Positive and negative harness controls** | Prove the gate passes on valid bundles and fails on seeded defective counterexamples. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.19` | **Seeded defect skill benchmarking** | Benchmark the testing skill against seeded defects in skill-eval-harness.ts. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.20` | **Held-out oracle evaluation** | Evaluate agent test submissions against independent held-out assertions. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.21` | **Anti-weakening and anti-skip detection** | Detect weakened assertions, deleted tests, .skip/.only, and fabricated reports automatically. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.22` | **False-pass and false-alarm measurement** | Track precision, recall, runtime, and cost across all 15 skill evaluation scenarios. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.23` | **Commit, shard, and attempt artifact binding** | Bind commit SHA, shard index, and attempt number into GateReconciliationReport. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.24` | **Reviewed snapshot and baseline updates** | Require explicit behavioral justification for any snapshot or visual baseline change. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.25` | **Fail-closed on timeout or cancellation** | Treat timed-out, cancelled, or missing shards as blocking deployment failures. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.26` | **Built-code coverage verification** | Verify tests exercise actual exported package modules and built Worker/Storybook bundles. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.27` | **Decision-boundary condition coverage** | Test boundary conditions across RBAC roles, word counts, SERP overlap, and budgets. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.28` | **Independent acceptance review** | Challenge implementation + test pairs with independent adversarial counterexamples. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.29` | **Protected-behavior deduplication** | Consolidate overlapping assertions into high-signal tests without losing failure localization. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
| `F15.30` | **Periodic full-audit selector calibration** | Compare selective execution against full workspace runs to verify zero missed regressions. | `meta_harness` | `required_ci` | `verified` | `packages/testing/src/nx-selector.ts, packages/testing/src/ci-gate.ts & packages/testing/src/skill-eval-harness.ts` |
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 323kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 323kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 323kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall