lidge-jun/codexclaw / plugins/codexclaw/skills/dev-testing
cxc-dev-testing
VerifiedUse for tests and release verification. Triggers: regression test, Playwright, E2E, contract test, coverage, CI flake, TDD, 테스트, 회귀 테스트, 품질 게이트.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.2 KB |
| agents/openai.yaml | config | 164 B |
| examples/console_logging.py | script | 1.1 KB |
| examples/element_discovery.py | script | 1.5 KB |
| examples/static_html_automation.py | script | 1.1 KB |
| references/acceptance-gates.md | markdown | 3.4 KB |
| references/acceptance-row-reachability.md | markdown | 1.1 KB |
| references/api-test-patterns.md | markdown | 3.4 KB |
| references/backend-testing.md | markdown | 4.3 KB |
| references/browser-testing.md | markdown | 3.5 KB |
| references/ci-integration.md | markdown | 1.3 KB |
| references/ci-pipeline.md | markdown | 8.5 KB |
| references/core/crud-test-matrix.md | markdown | 2.2 KB |
| references/edge-first-testing.md | markdown | 2.5 KB |
| references/limited-oracle-evaluation.md | markdown | 2.7 KB |
| references/load-testing.md | markdown | 6.9 KB |
| references/ml-evaluation.md | markdown | 7.2 KB |
| references/patch-integrity.md | markdown | 1.4 KB |
| references/property-and-mutation-testing.md | markdown | 652 B |
| references/tdd-and-regressions.md | markdown | 5.4 KB |
| references/test-oracle-integrity.md | markdown | 2.0 KB |
| references/test-strategy.md | markdown | 1.5 KB |
| scripts/with_server.py | script | 3.6 KB |
references/ci-pipeline.md
View on GitHub# CI Pipeline Templates
> Deep reference for `dev-testing` §5 CI Pipeline Integration.
## 1. Node.js Workflow Template
```yaml
name: node-test
on:
push:
pull_request:
concurrency:
group: node-test-${{ github.ref }}
cancel-in-progress: true
jobs:
quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22, cache: npm }
- run: npm ci
- run: npm run lint
- run: npx tsc --noEmit
test:
needs: quality
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix: { node-version: [22, 24], shard: [1, 2, 3] }
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: ${{ matrix.node-version }}, cache: npm }
- run: npm ci
- run: npx vitest run --coverage --shard=${{ matrix.shard }}/3
- run: npm run test:contract
e2e:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22, cache: npm }
- run: npm ci
- run: npx playwright install --with-deps chromium
- run: npx playwright test
```
## 2. Python Workflow Template
```yaml
name: python-test
on:
push:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- run: pip install -r requirements-dev.txt
- run: pytest --cov --cov-report=xml -n auto
- run: pytest tests/contracts -q
- run: pip-audit --strict
```
## 3. Matrix / Parallelization
| Axis | Include When | Example |
|------|--------------|---------|
| runtime versions | library or SDK compatibility matters | Node 22/24, Python 3.11-3.13 |
| OS | native modules or CLI behavior matter | ubuntu + macOS |
| shards | suites exceed CI budget | `1/4..4/4` |
```bash
npx vitest run --shard=2/4
npx playwright test --shard=2/4 --workers=4
pytest -n auto --dist=loadgroup
```
## 4. Coverage Reporting Integration
- publish `lcov.info` or `coverage.xml`
- upload junit / XML results for annotations
- keep contract reports separate from unit coverage
- fail the build when thresholds or diff coverage drop
## 5. Flaky Test Policy (canonical — `TEST-FLAKE-*`)
Canonical owner: `dev-testing`. Other skills carry pointer stubs only (see
`dev` `references/skill-ownership.md`). `dev-debugging` owns the diagnostic
method; this section owns the policy — what CI may do about a flake, and what
counts as closing one.
A flake is a defect that has not been diagnosed yet. Mechanisms that make CI
green without diagnosing it ship that defect.
### 5.1 Eliminate the nondeterminism (`TEST-FLAKE-ELIMINATE-01`, STRICT)
A flaky test is a defect in the test or in the code under test. Diagnose the
source of nondeterminism and remove it. **A flake is closed when the cause is
named, not when the suite is green.**
This is a CLOSURE rule: it governs the claim "this flake is fixed", not the
question of whether work may proceed meanwhile. A quarantine under §5.3 does not
violate it, because a quarantine explicitly does not close the defect.
| Signal | Cause to remove |
|--------|-----------------|
| intermittent timeout | implicit timing — wait on an observable condition or a fake clock |
| passes locally, fails in CI | an unpinned seed, an uncontainerized dependency, or an implicit wait |
| order-dependent failure | shared mutable state or fixture leakage between tests |
| CI-only HTTP failure | a live network dependency |
| snapshot variance | unpinned fonts, time, locale, or dynamic regions |
| passes alone, fails in the suite | resource contention or global state — isolate, then fix the sharing |
After a fix, search for other tests with the same missing cleanup or the same
timing assumption. One cause commonly has siblings, and closing only the
instance that failed leaves the rest to fail later.
### 5.2 Re-running is not a resolution (`TEST-FLAKE-RERUN-01`, STRICT)
Re-running a failed job or test to obtain green is **not** a fix and is never
recorded as one. Raising a timeout until a test passes is the same violation
wearing a config change.
A re-run is permitted for exactly one purpose: measuring the failure RATE as
diagnostic input. When you use it that way, write the measurement down — "4 of 5
runs, 4 different tests" is evidence; "passed on retry" is not.
This is the CI-facing half of `TEST-ANTI-FLAKE-01` (`dev-testing` SKILL.md §1.5)
and `TEST-CI-GREEN-01` (`dev-testing` SKILL.md §5.5).
### 5.3 Quarantine is an exception with a cost (`TEST-FLAKE-QUARANTINE-01`, DEFAULT)
Quarantine is permitted only when the flake blocks delivery of work that does not
depend on the code under test — state which delivery, and why it is independent —
AND all four of these are recorded in the same change (**STRICT**: the four
fields are not waivable; only the decision to defer is DEFAULT):
1. the exact test name,
2. a named owner,
3. a removal deadline, and
4. the suspected cause.
A quarantine without a deadline is a deletion with extra steps. Quarantine never
closes the defect — it defers it, and the deadline is the receipt.
A quarantine carrying all four fields is the one form of `.skip()` that is not a
`TEST-PATCH-INTEGRITY-01` red flag (`dev-testing` SKILL.md §8). An undocumented
skip still is.
### 5.4 "Environmental" is a claim, not an observation (`TEST-FLAKE-ATTRIBUTION-01`, DEFAULT)
Before calling a failure environmental or pre-existing, read
[Baseline versus defect](../../dev-devops/references/ci-cd-deploy.md#62-baseline-versus-defect-devops-baseline-defect-01-strict)
(DEVOPS-BASELINE-DEFECT-01). This test-side attribution rule applies the owner’s proof,
and it is DEFAULT rather than STRICT for one reason only: step 3 sometimes needs
CI access an agent does not have. In that case record the gap. **A recorded gap
is not a waiver** — the failure remains a candidate defect, and the claim
"environmental" remains unmade.
**When both rules apply, the STRICT one governs.** A release or freeze decision
is covered by the [strict release owner](../../dev-devops/references/ci-cd-deploy.md#62-baseline-versus-defect-devops-baseline-defect-01-strict), so an agent cannot reach the
weaker class by loading only `dev-testing`.
### 5.5 Counting greens
How many consecutive green runs make a flaky-capable suite trustworthy is a
release-gate question, not a remediation one: `DEVOPS-FLAKE-STABILITY-01` in
`dev-devops` `references/ci-cd-deploy.md` §6.5.
## 6. Recommended Job Order
```text
quality
→ backend / unit / integration
→ contract
→ e2e
→ security
→ coverage aggregation
```
## 7. Release Runtime vs Compatibility Matrix
Use the release runtime required by the publish mechanism for release jobs. For
example, npm Trusted Publishing requires a modern Node/npm pair, so release
examples should use Node 22 unless the project has a newer declared baseline.
Keep compatibility testing separate from release publishing:
| Matrix | Purpose | Example |
|---|---|---|
| Release runtime | Runtime used to build and publish artifacts | Node 22 for npm Trusted Publishing |
| Compatibility runtime | Extra supported versions the package promises to run on | Node 20 only when a project explicitly carries legacy/EOL support |
| OS smoke | Platform behavior for native modules, CLIs, installers, or shell shims | ubuntu, macOS, Windows for declared support |
Do not put legacy/EOL runtimes in generic release examples. If a project still
supports Node 20, label that lane as legacy compatibility and keep publish jobs
on the release runtime required by the registry or deployment target.
## 8. CLI Release Smoke Matrix
For CLI packages or native modules, matrix only the platforms the project
declares or the release claims to support. A Linux-only package does not need a
Windows smoke, but a cross-platform CLI claim needs runner evidence for each
declared platform.
Minimum smoke:
1. Install the packed artifact or published package.
2. Resolve the binary path.
3. Run `--version`.
4. Run `--help`.
5. Run one safe non-destructive command that touches platform-sensitive paths
when the release depends on those paths.
CI matrix proof is enough for pure library/import behavior. For desktop,
installer, profile, permission, PATH, shell-shim, or visible OS behavior that CI
cannot observe, hand off to `dev-devops/references/cross-platform-release.md`.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |