netresearch/typo3-testing-skill / skills/typo3-testing
typo3-testing
FlaggedUse when reproducing defects as failing tests, testing template or rendered output changes, confirming changes on a running TYPO3 instance, or setting up TYPO3 extension tests: unit/functional/E2E, PHPUnit 11-13, mutation testing, final-class mocks (v14), CI matrix across TYPO3 12/13/14.3 LTS, typo3-ci-workflows meta-package, CI failures. Also: transaction rollback, worker crashes, competing requests, testing-framework setup, test matrix, integration/e2e testing, coverage, test generation.
checkpoints.yaml
View on GitHub# SPDX-License-Identifier: MIT
# SPDX-FileCopyrightText: Netresearch DTT GmbH
# Checkpoints for typo3-testing skill
# Focuses on PHPUnit, test structure, CI, PHPStan, and code coverage
#
# Supports two CI patterns:
# 1. Traditional: local Build/UnitTests.xml + runTests.sh + direct action usage
# 2. Shared workflow: netresearch/typo3-ci-workflows (reusable CI, handles
# PHPUnit config naming, SHA pinning, coverage, and PHPStan internally)
version: 1
skill_id: typo3-testing
# Only run for TYPO3 extensions: the checks assume extension test infrastructure
# (runTests.sh, Build/ PHPUnit configs, typo3-ci-workflows). Without this gate
# /assess ran them against a Python repository and reported 45 failures.
preconditions:
- type: json_path
target: composer.json
pattern: '.type == "typo3-cms-extension"'
mechanical:
# === SHARED CI WORKFLOW DETECTION ===
# Check first — many later checks can be relaxed when a shared CI workflow
# handles PHPUnit config, SHA pinning, coverage, and PHPStan.
- id: TT-00
type: regex
target: .github/workflows/*.yml
pattern: 'uses:\s*netresearch/typo3-ci-workflows'
severity: info
desc: "Detects netresearch/typo3-ci-workflows reusable workflow (relaxes TT-01..05, TT-21, TT-24)"
# === PHPUNIT CONFIGURATION ===
# TYPO3 testing-framework convention: Build/UnitTests.xml + Build/FunctionalTests.xml
# netresearch/typo3-ci-workflows also accepts: Build/phpunit.xml + Build/phpunit.functional.xml
# Extensions that keep every PHPUnit config in one subdirectory use
# Build/phpunit/UnitTests.xml + Build/phpunit/FunctionalTests.xml.
# All three layouts are valid; TT-01/02/04/05/06 accept each of them.
# When TT-00 passes, either naming convention is valid.
- id: TT-01
type: file_exists
target: "{Build/UnitTests.xml,Build/phpunit.xml,Build/phpunit/UnitTests.xml}"
severity: error
desc: "PHPUnit unit test config must exist (Build/UnitTests.xml, Build/phpunit.xml or Build/phpunit/UnitTests.xml)"
- id: TT-02
type: file_exists
target: "{Build/FunctionalTests.xml,Build/phpunit.functional.xml,Build/phpunit/FunctionalTests.xml}"
severity: error
desc: "PHPUnit functional test config must exist (Build/FunctionalTests.xml, Build/phpunit.functional.xml or Build/phpunit/FunctionalTests.xml)"
- id: TT-03
type: file_not_exists
target: phpunit.xml
severity: warning
desc: "phpunit.xml in project root is discouraged; use Build/ directory instead"
- id: TT-04
type: regex
target: "{Build/UnitTests.xml,Build/phpunit.xml,Build/phpunit/UnitTests.xml}"
pattern: '<testsuite.*name="unit"'
severity: error
desc: "PHPUnit config must define unit test suite"
- id: TT-05
type: regex
target: "{Build/FunctionalTests.xml,Build/phpunit.functional.xml,Build/phpunit/FunctionalTests.xml}"
pattern: '<testsuite.*name="functional"'
severity: error
desc: "PHPUnit config must define functional test suite"
# === PHPUNIT DEPRECATED ATTRIBUTES ===
- id: TT-06
type: regex_not
target: "{Build/FunctionalTests.xml,Build/phpunit.xml,Build/UnitTests.xml,Build/phpunit.functional.xml,Build/phpunit/UnitTests.xml,Build/phpunit/FunctionalTests.xml}"
pattern: 'displayDetailsOnIncompleteTests|displayDetailsOnSkippedTests'
severity: warning
desc: "PHPUnit configs must not use displayDetailsOnIncompleteTests or displayDetailsOnSkippedTests (removed in PHPUnit 12)"
# === TEST DIRECTORY STRUCTURE ===
- id: TT-10
type: file_exists
target: Tests/Unit/
severity: error
desc: "Tests/Unit/ directory must exist"
- id: TT-11
type: file_exists
target: Tests/Functional/
severity: warning
desc: "Tests/Functional/ directory should exist"
- id: TT-12
type: regex
target: Tests/Unit/**/*Test.php
pattern: 'class.*Test.*extends.*TestCase'
severity: error
desc: "Unit tests must extend TestCase"
- id: TT-13
type: regex
target: Tests/Functional/**/*Test.php
pattern: 'class.*Test.*extends.*FunctionalTestCase'
severity: warning
desc: "Functional tests should extend FunctionalTestCase"
# === CI WORKFLOW WITH TEST EXECUTION ===
- id: TT-20
type: file_exists
target: .github/workflows/ci.yml
severity: warning
desc: "CI workflow should exist (ci.yml, ci.yaml, or tests.yml)"
- id: TT-21
type: regex
target: .github/workflows/*.yml
pattern: 'phpunit|composer.*test|runTests|typo3-ci-workflows'
severity: error
desc: "CI workflow must execute tests (directly or via reusable workflow)"
- id: TT-22
type: regex
target: .github/workflows/*.yml
pattern: 'php-version.*\[.*8\.[2-9]|php-versions'
severity: warning
desc: "CI should test multiple PHP versions (directly or via php-versions input)"
- id: TT-23
type: regex
target: .github/workflows/*.yml
pattern: 'typo3.*\[.*1[3-4]|typo3-versions'
severity: warning
desc: "CI should test multiple TYPO3 versions"
- id: TT-24
type: regex
target: .github/workflows/*.yml
pattern: 'uses:.*@[a-f0-9]{40}|uses:\s*netresearch/typo3-ci-workflows'
severity: error
desc: "GitHub Actions must be pinned to SHA (or use trusted reusable workflow)"
# === PHPSTAN CONFIGURATION ===
- id: TT-30
type: file_exists
target: "{phpstan.neon,Build/phpstan.neon,Build/phpstan/phpstan.neon}"
severity: warning
desc: "phpstan.neon configuration should exist"
- id: TT-31
type: file_exists
target: "{phpstan-baseline.neon,Build/phpstan-baseline.neon,Build/phpstan/phpstan-baseline.neon}"
severity: info
desc: "phpstan-baseline.neon for gradual adoption (root or Build/ subdir)"
- id: TT-32
type: regex
target: "{phpstan.neon,Build/phpstan.neon,Build/phpstan/phpstan.neon}"
pattern: 'level:\s*(10|max)'
severity: warning
desc: "PHPStan must use level 10 (max) — this is the strict check for projects using typo3-ci-workflows"
- id: TT-33
type: regex
target: "{phpstan.neon,Build/phpstan.neon,Build/phpstan/phpstan.neon}"
pattern: "typo3|TYPO3"
severity: info
desc: "PHPStan should include TYPO3-specific configuration"
- id: TT-34
type: regex
target: .github/workflows/*.yml
pattern: 'phpstan|composer.*stan|composer.*analyze|typo3-ci-workflows'
severity: warning
desc: "CI should run PHPStan analysis (directly or via reusable workflow)"
# Was a single-line `command:` chaining four alternatives with `||`, which the
# runner rejects outright (command-chaining metacharacter), so it never ran and
# reported a failure on every project. It also relied on an `expected:` field the
# runner does not parse — the verdict has to be the exit status. Both are fixed
# here by moving the logic into a screened script body.
#
# The level is read with `grep -oE` plus a bash suffix strip, not `grep -oP
# …\K`: where PCRE is missing (BSD grep), the PCRE spelling captured nothing
# from every candidate file and the loop then reported "no PHPStan
# configuration declaring a level" — a false failure against a conformant
# project on every such machine.
#
# Every candidate configuration is inspected before the verdict. A project
# with Build/phpstan.neon at level 9 and phpstan.neon at level 10 satisfies
# the rule; exiting at the first file below 10 reported it as non-conformant.
- id: TT-35
type: script
command: |
bad=""
for f in Build/phpstan.neon Build/phpstan/phpstan.neon phpstan.neon phpstan.neon.dist; do
[ -f "$f" ] || continue
lvl=$(grep -m1 -oE 'level:[[:space:]]*([0-9]+|max)' "$f" 2>/dev/null)
lvl="${lvl##*[[:space:]:]}"
[ -z "$lvl" ] && continue
if [ "$lvl" = "max" ] || [ "$lvl" = "10" ]; then
exit 0
fi
bad="$bad $f=$lvl"
done
if [ -n "$bad" ]; then
echo "FAIL: no PHPStan configuration declares level 10 or max; found:$bad"
exit 1
fi
echo "FAIL: no PHPStan configuration declaring a level was found"
exit 1
severity: warning
desc: "PHPStan must use level 10 (max) — this is the strict check for projects using typo3-ci-workflows"
# === CODE COVERAGE ===
- id: TT-40
type: file_exists
target: codecov.yml
severity: info
desc: "codecov.yml configuration for coverage settings"
- id: TT-41
type: regex
target: .github/workflows/*.yml
pattern: 'codecov|upload-coverage|typo3-ci-workflows'
severity: warning
desc: "CI should upload coverage (directly or via reusable workflow with upload-coverage: true)"
- id: TT-42
type: regex
target: .github/workflows/{ci,tests}.{yml,yaml}
pattern: 'coverage.*clover|XDEBUG_MODE.*coverage|--coverage|upload-coverage:\s*true'
severity: warning
desc: "CI should generate code coverage"
- id: TT-43
type: contains
target: README.md
pattern: "codecov.io"
severity: info
desc: "README should display Codecov badge"
# === RUNTESTS.SH SCRIPT (mandatory for all Netresearch TYPO3 extensions) ===
# TT-51/52/53/57/59 read the shared runner when the extension only carries
# the stub. The Netresearch standard Build/Scripts/runTests.sh is a stub
# (typo3-ci-workflows assets/Build/Scripts/runTests.stub.sh) that only execs
# .Build/bin/runTests.sh, so grepping the stub measured nothing. That bin
# file is not the runner either: it is Composer's ~1 KB bin proxy, which
# execs the package file. When the stub is detected, the checks read the
# package file directly, at <vendor-dir from composer.json, default
# "vendor">/netresearch/typo3-ci-workflows/assets/Build/Scripts/runTests.sh.
# When that file is absent (a clean checkout), the check passes by
# delegation as long as composer.json requires
# netresearch/typo3-ci-workflows, and fails when the stub has nothing to
# hand over to.
- id: TT-50
type: file_exists
target: Build/Scripts/runTests.sh
severity: error
desc: "Build/Scripts/runTests.sh must exist as the unified test runner for all TYPO3 extensions"
- id: TT-51
type: script
command: |
f=Build/Scripts/runTests.sh
[ -f "$f" ] || exit 1
if grep -q 'typo3-ci-workflows' "$f" && grep -qE '^exec[[:blank:]]+"[$][{]?RUNNER[}]?"' "$f"; then
vd=$(jq -r '.config["vendor-dir"] // "vendor"' composer.json 2>/dev/null) || vd=vendor
f="${vd:-vendor}/netresearch/typo3-ci-workflows/assets/Build/Scripts/runTests.sh"
if [ ! -f "$f" ]; then
jq -e '((.require // {}) + (."require-dev" // {})) | has("netresearch/typo3-ci-workflows")' composer.json >/dev/null 2>&1
exit $?
fi
fi
grep -qE 'TYPO3_VERSION|typo3/core' "$f"
severity: warning
desc: "runTests.sh should support TYPO3 version selection"
- id: TT-52
type: script
command: |
f=Build/Scripts/runTests.sh
[ -f "$f" ] || exit 1
if grep -q 'typo3-ci-workflows' "$f" && grep -qE '^exec[[:blank:]]+"[$][{]?RUNNER[}]?"' "$f"; then
vd=$(jq -r '.config["vendor-dir"] // "vendor"' composer.json 2>/dev/null) || vd=vendor
f="${vd:-vendor}/netresearch/typo3-ci-workflows/assets/Build/Scripts/runTests.sh"
if [ ! -f "$f" ]; then
jq -e '((.require // {}) + (."require-dev" // {})) | has("netresearch/typo3-ci-workflows")' composer.json >/dev/null 2>&1
exit $?
fi
fi
grep -qE 'PHP_VERSION|php.*version' "$f"
severity: warning
desc: "runTests.sh should support PHP version selection (-p flag)"
- id: TT-53
type: script
command: |
f=Build/Scripts/runTests.sh
[ -f "$f" ] || exit 1
if grep -q 'typo3-ci-workflows' "$f" && grep -qE '^exec[[:blank:]]+"[$][{]?RUNNER[}]?"' "$f"; then
vd=$(jq -r '.config["vendor-dir"] // "vendor"' composer.json 2>/dev/null) || vd=vendor
f="${vd:-vendor}/netresearch/typo3-ci-workflows/assets/Build/Scripts/runTests.sh"
if [ ! -f "$f" ]; then
jq -e '((.require // {}) + (."require-dev" // {})) | has("netresearch/typo3-ci-workflows")' composer.json >/dev/null 2>&1
exit $?
fi
fi
grep -qE 'suite|SUITE' "$f"
severity: warning
desc: "runTests.sh should support test suite selection (-s flag)"
- id: TT-54
type: command
command: 'test -x Build/Scripts/runTests.sh'
severity: error
desc: "Build/Scripts/runTests.sh must be executable"
# Two valid entry-point layouts, and this check accepts both:
# 1. composer scripts call Build/Scripts/runTests.sh directly;
# 2. composer scripts (ci:test:php:unit, ci:test:php:functional, ...) call
# phpunit directly, and runTests.sh runs the suites in TYPO3
# core-testing containers for multi-version runs. The TYPO3 reference
# extension TYPO3BestPractices/tea does exactly this (check:tests:unit
# calls phpunit; runTests.sh starts .Build/bin/phpunit in a container),
# and 17 of 25 netresearch/t3x-* extensions did on 2026-10-07. tea names
# its scripts check:tests:unit / check:tests:functional, so those count too.
# runTests.sh itself stays mandatory (TT-50) either way.
# Requiring shape 1 alone reported an error-free repo as non-conformant.
# A composer.json with neither shape still fails: it defines no test entry
# point at all.
#
# Read structurally out of `.scripts`, not as a regex over the whole file:
# `scripts-descriptions` is a real composer field and NR extensions document
# exactly these script names in it, so the text match passed a composer.json
# whose `scripts` object defines no test entry point.
- id: TT-55
type: json_path
target: composer.json
pattern: '(.scripts // {}) | (has("ci:test:php:unit") or has("ci:test:php:functional") or has("check:tests:unit") or has("check:tests:functional") or (tostring | contains("Build/Scripts/runTests.sh")))'
severity: warning
desc: "composer.json should define a test entry point — either a script invoking Build/Scripts/runTests.sh, or ci:test:php:unit / ci:test:php:functional scripts, or check:tests:unit / check:tests:functional scripts (TYPO3BestPractices/tea)"
- id: TT-56
type: regex
target: Makefile
pattern: 'typo3-ci-workflows/Makefile.include'
severity: info
desc: "Makefile should include shared targets from typo3-ci-workflows"
# The TYPO3 standard build layout is .Build/ (TYPO3BestPractices/tea:
# vendor-dir .Build/vendor, bin-dir .Build/bin, web-dir .Build/public), and
# runTests.sh, the CI templates and the PHPUnit configs in assets/ assume it.
# Lowercase .build/vendor and an unset vendor-dir are legacy: a warning, not
# an error, so existing extensions keep passing until their build setup is
# touched. typo3-conformance checks the same value as TC-194.
- id: TT-137
type: json_path
target: composer.json
pattern: '.config["vendor-dir"] == ".Build/vendor"'
severity: warning
desc: "composer.json config.vendor-dir should be .Build/vendor (TYPO3 standard, see TYPO3BestPractices/tea); .build/vendor or an unset vendor-dir is a legacy layout - migrate when touching the build setup"
# === RUNTESTS.SH QUALITY ===
- id: TT-57
type: script
command: |
f=Build/Scripts/runTests.sh
[ -f "$f" ] || exit 1
if grep -q 'typo3-ci-workflows' "$f" && grep -qE '^exec[[:blank:]]+"[$][{]?RUNNER[}]?"' "$f"; then
vd=$(jq -r '.config["vendor-dir"] // "vendor"' composer.json 2>/dev/null) || vd=vendor
f="${vd:-vendor}/netresearch/typo3-ci-workflows/assets/Build/Scripts/runTests.sh"
if [ ! -f "$f" ]; then
jq -e '((.require // {}) + (."require-dev" // {})) | has("netresearch/typo3-ci-workflows")' composer.json >/dev/null 2>&1
exit $?
fi
fi
grep -qE 'ghcr\.io/typo3/core-testing|CONTAINER_BIN|docker\s+run' "$f"
severity: warning
desc: "runTests.sh should use Docker containers (TYPO3 core-testing images) for consistent test environments"
# Fires only on the real bug: a make RECIPE that invokes runTests.sh without
# -s. It no longer demands that the Makefile drive runTests.sh at all, which
# reported an error against every repo where the Makefile is a thin wrapper
# over composer scripts (ci:test:php:unit, ...) and runTests.sh is reserved
# for Docker-based multi-version runs.
#
# Anchored on a literal TAB: make recipes must be tab-indented, assignments
# never are, so `RUNTESTS := Build/Scripts/runTests.sh` — with or without the
# leading spaces it gets inside an ifeq block — is not mistaken for a call.
# The runner must also be the recipe's COMMAND WORD, after make's optional
# `@`/`-`/`+` prefixes: a `help:` target whose `@echo` merely names
# runTests.sh is a mention, not a call, and matching it anywhere on the line
# reported an error against a correct Makefile. `[)}]` accepts both
# $(RUNTESTS) and ${RUNTESTS}; `\S*runTests\.sh` accepts the direct path.
# The lookahead keeps this a single grep: a `grep … | grep -q …` pipeline
# reports the opposite verdict under `set -o pipefail`, because grep -q
# closes the pipe early and the SIGPIPE on the left-hand grep becomes the
# pipeline's status.
# Known limitations: a call split across a backslash line-continuation (also
# true of the regex this replaced), and a call that is not in command
# position (`cd Build && $(RUNTESTS) unit`), are not seen. Needs grep -P:
# where PCRE is missing, `grep -qP` exits 2, the leading `!` inverts that and
# the check passes — it fails open instead of reporting every Makefile as
# broken. TT-35 and TT-105 do not rely on that; they were switched to a
# PCRE-free extraction, because their PCRE spelling failed CLOSED.
- id: TT-58
type: command
pattern: "! grep -qP '^\\t[@+-]*([$][({]RUNTESTS[)}]|\\S*runTests\\.sh)(?!.* -s[ =])' Makefile 2>/dev/null"
severity: error
desc: "Makefile recipes that invoke runTests.sh must pass the -s flag (e.g. $(RUNTESTS) -s unit, not $(RUNTESTS) unit). A Makefile that wraps composer scripts instead, or no Makefile at all, satisfies this check."
- id: TT-59
type: script
command: |
f=Build/Scripts/runTests.sh
[ -f "$f" ] || exit 1
if grep -q 'typo3-ci-workflows' "$f" && grep -qE '^exec[[:blank:]]+"[$][{]?RUNNER[}]?"' "$f"; then
vd=$(jq -r '.config["vendor-dir"] // "vendor"' composer.json 2>/dev/null) || vd=vendor
f="${vd:-vendor}/netresearch/typo3-ci-workflows/assets/Build/Scripts/runTests.sh"
if [ ! -f "$f" ]; then
jq -e '((.require // {}) + (."require-dev" // {})) | has("netresearch/typo3-ci-workflows")' composer.json >/dev/null 2>&1
exit $?
fi
fi
grep -qE 'NETWORK=.*\$\{?SUFFIX' "$f"
severity: warning
desc: "Docker network name should include per-run suffix to avoid conflicts with concurrent runs"
# === COMPOSER.JSON TEST DEPENDENCIES ===
# TT-60 removed: phpunit/phpunit is pulled transitively via typo3/testing-framework.
# Requiring it directly breaks the PHP-8.2 matrix cell because phpunit 12.5.8+
# requires PHP ≥ 8.3. Do NOT add a direct phpunit/phpunit require-dev entry.
# `||`-chained on one line, so the runner rejected it unread and it failed on
# every project, including ones that do have PHPStan.
#
# The composer.json branch reads declared dependency KEYS. `grep -q phpstan
# composer.json` also passed on a `description` or `scripts-descriptions`
# entry that merely names PHPStan, so the warning was suppressed for projects
# that never install it. netresearch/typo3-ci-workflows counts as a key,
# because it is the shared CI package this check means by "transitively".
- id: TT-61
type: script
command: |
[ -f .Build/bin/phpstan ] && exit 0
[ -f vendor/bin/phpstan ] && exit 0
jq -e '((.require // {}) + (."require-dev" // {})) | keys | any(test("phpstan") or . == "netresearch/typo3-ci-workflows")' composer.json >/dev/null 2>&1 && exit 0
grep -rq phpstan .Build/vendor/composer/installed.json 2>/dev/null && exit 0
grep -rq phpstan vendor/composer/installed.json 2>/dev/null && exit 0
echo "FAIL: PHPStan is not available directly or transitively"
exit 1
severity: warning
desc: "PHPStan should be available (directly or transitively via a shared CI package)"
- id: TT-62
type: json_path
target: composer.json
pattern: '.scripts.test // .scripts["ci:test"] // .scripts["test:unit"] // .scripts["ci:test:php:unit"] // .scripts["check:tests:unit"]'
severity: warning
desc: "composer.json should define test script (test, ci:test, test:unit, ci:test:php:unit, or check:tests:unit as in TYPO3BestPractices/tea)"
# === ARCHITECTURE TESTING ===
- id: TT-14
type: file_exists
target: Tests/Architecture/
severity: info
desc: "Tests/Architecture/ directory should exist for phpat architecture tests"
- id: TT-15
type: json_path
target: composer.json
pattern: '.["require-dev"]["phpat/phpat"] // .["require-dev"]["netresearch/typo3-ci-workflows"]'
severity: warning
desc: "composer.json should require phpat/phpat for architecture testing (package was renamed from carlosas/phpat in 2023; also satisfied when netresearch/typo3-ci-workflows meta-package is present, as phpat is transitive)"
# === E2E TESTING (Playwright) ===
- id: TT-16
type: file_exists
target: "{playwright.config.ts,playwright.config.js,Build/playwright.config.ts,Tests/E2E/playwright.config.ts}"
severity: info
desc: "Playwright config should exist for E2E testing (root, Build/, or Tests/E2E/)"
- id: TT-17
type: file_exists
target: "{package.json,Build/package.json,Tests/E2E/package.json,Tests/E2E/Playwright/package.json}"
severity: info
desc: "package.json should exist for Playwright/JS test dependencies (root or Build/ or Tests/E2E/ subpath)"
- id: TT-17a
type: command
command: |
cfg=""
for f in Build/playwright.config.ts playwright.config.ts; do
[ -f "$f" ] && cfg="$f" && break
done
[ -z "$cfg" ] && exit 0
grep -q 'chromium' "$cfg" && grep -q 'firefox' "$cfg" && exit 0
echo "FAIL: $cfg should define at least Chromium and Firefox projects"
exit 1
severity: warning
desc: "If playwright.config.ts exists, it should define at least Chromium and Firefox projects for cross-browser E2E coverage"
# A suite that exists but is never invoked is not a gate, and it reads like one
# in a release note. nr_passkeys_fe 1.0.0 shipped 19 specifications that every
# file skipped and no workflow ran.
- id: TT-17c
type: command
command: |
specs=$(find Tests/E2E -name '*.spec.ts' 2>/dev/null)
[ -z "$specs" ] && exit 0
# An executable invocation, not a mention: a `run:` line that starts the
# suite, or a caller of a shared e2e workflow. A cache key or a comment
# naming playwright is not a run.
grep -rhE '^\s*(-?\s*(run|uses):).*(npx playwright|playwright test|runTests\.sh -s e2e|ci-e2e|/e2e\.ya?ml@)' \
.github/workflows/ 2>/dev/null | grep -q . && exit 0
echo "FAIL: Tests/E2E holds specifications but no workflow invokes them"
exit 1
severity: error
desc: "If Tests/E2E holds spec files, a workflow must invoke the suite — an uninvoked suite is not a gate"
- id: TT-17d
type: command
command: |
specs=$(find Tests/E2E -name '*.spec.ts' 2>/dev/null)
[ -z "$specs" ] && exit 0
# Column 0 is what separates a file-level skip from a conditional one
# inside a test body, which is legitimate and always indented.
skipped=$(printf '%s\n' "$specs" | xargs grep -lE '^(test\.skip|test\.describe\.skip|describe\.skip)\(' 2>/dev/null | wc -l)
total=$(printf '%s\n' "$specs" | wc -l)
[ "$skipped" -eq 0 ] && exit 0
echo "FAIL: $skipped of $total E2E spec files are disabled by a file-level skip"
exit 1
severity: error
desc: "No E2E spec file may be disabled by a file-level test.skip()/test.describe.skip() — a skipped suite passes CI while testing nothing"
# === MUTATION TESTING ===
- id: TT-18
type: file_exists
target: "{infection.json5,infection.json.dist,infection.json,Build/infection.json5,Build/infection.json.dist}"
severity: info
desc: "Infection config should exist for mutation testing (accepts .json5, .json.dist, .json)"
- id: TT-18a
type: command
command: |
test -f infection.json5 || exit 0
grep -rql 'infection\|mutation\|ci:test:php:mutation' .github/workflows/*.yml 2>/dev/null && exit 0
echo "FAIL: infection.json5 exists but mutation testing is not configured in CI"
exit 1
severity: warning
desc: "If infection.json5 exists, mutation testing should be configured in CI (at least as a scheduled job)"
# === ADDITIONAL DEV DEPENDENCIES ===
# Accept either a direct require-dev declaration OR transitive availability
# via the netresearch/typo3-ci-workflows meta-package, which bundles
# php-cs-fixer, typo3/coding-standards, rector, phpstan, infection, etc.
# Same pattern as TT-15 (phpat) — see references/ci-workflows-meta-package.md
- id: TT-63
type: json_path
target: composer.json
pattern: '.["require-dev"]["friendsofphp/php-cs-fixer"] // .["require-dev"]["typo3/coding-standards"] // .["require-dev"]["netresearch/typo3-ci-workflows"]'
severity: warning
desc: "composer.json should require php-cs-fixer or typo3/coding-standards in require-dev (also satisfied transitively via netresearch/typo3-ci-workflows)"
- id: TT-64
type: json_path
target: composer.json
pattern: '.["require-dev"]["rector/rector"] // .["require-dev"]["netresearch/typo3-ci-workflows"]'
severity: info
desc: "composer.json should require rector/rector in require-dev (also satisfied transitively via netresearch/typo3-ci-workflows)"
# === CI MATRIX BEST PRACTICES ===
- id: TT-25
type: regex
target: .github/workflows/*.yml
pattern: 'fail-fast:\s*false'
severity: info
desc: "CI matrix should use fail-fast: false to run all combinations"
# === TEST ENVIRONMENT GUARDS ===
- id: TT-75
type: regex
target: Tests/**/*Test.php
pattern: 'tearDown\s*\(\s*\)\s*:\s*void'
severity: warning
desc: "Test classes creating filesystem artifacts (tempDir, tempFile) must implement tearDown() for cleanup"
note: "Check manually: only flag when setUp() creates files/dirs but tearDown() is missing"
# Was `type: regex` — it FAILED every test suite that does not use the
# attribute and passed the ones that do. The attribute exists from PHPUnit
# 12.5.2 on; PHPUnit 11.5 skips attributes whose class does not exist
# (Metadata/Parser/AttributeParser.php, `if (!class_exists(...)) continue`),
# so on 11.5 it has no effect, it is not fatal. Kept as an info-level hint
# towards createStub(), the approach references/unit-testing.md prefers.
- id: TT-76
type: regex_not
target: Tests/**/*Test.php
pattern: 'AllowMockObjectsWithoutExpectations'
severity: info
provenance: upstream
source: https://github.com/sebastianbergmann/phpunit/blob/12.5.2/src/Framework/Attributes/AllowMockObjectsWithoutExpectations.php
verified: 2026-09-27
desc: "Prefer createStub() for test doubles without expectations: #[AllowMockObjectsWithoutExpectations] exists only from PHPUnit 12.5.2 and has no effect on PHPUnit 11.5"
# === NEGATIVE CHECKS (should NOT exist) ===
- id: TT-90
type: regex_not
target: .github/workflows/*.yml
pattern: 'ddev start|ddev exec|ddev-github-action|setup-ddev'
severity: error
desc: "DDEV must not be used in CI/CD. Use PHP built-in server or Docker containers."
tags: [ci, ddev, e2e]
# === GIT HOOKS ===
# Netresearch convention: keep testing/CI config under Build/ so the repo
# root stays focused on end-user files (README, LICENSE, composer.json,
# ext_emconf.php). captainhook/hook-installer reads the config path from
# composer.json "extra.captainhook.config", so Build/captainhook.json is
# the enforced default.
- id: TT-71
type: file_exists
target: "{Build/captainhook.json,captainhook.json,.captainhook/captainhook.json}"
severity: error
desc: >-
captainhook config must exist for local git hook automation (pre-commit,
commit-msg, pre-push) — mandatory for Netresearch extensions.
Netresearch default: Build/captainhook.json (declared in composer.json
"extra.captainhook.config"). Root captainhook.json or
.captainhook/captainhook.json are accepted for backward compatibility
but root location is flagged by TT-71a.
- id: TT-71a
type: file_not_exists
target: captainhook.json
severity: warning
desc: >-
captainhook.json should live under Build/captainhook.json (Netresearch
convention — keeps repo root clean). Move to Build/ and declare
"extra": {"captainhook": {"config": "Build/captainhook.json"}} in
composer.json so captainhook/hook-installer picks it up.
- id: TT-71b
type: json_path
target: composer.json
pattern: '.extra.captainhook.config == "Build/captainhook.json"'
severity: warning
desc: >-
composer.json should declare extra.captainhook.config:
"Build/captainhook.json" so the captainhook/hook-installer plugin
installs hooks using the Build/ subdir config on every composer install.
# === GITIGNORE CHECKS ===
- id: TT-70
type: contains
target: .gitignore
pattern: "composer.lock"
severity: error
desc: "composer.lock should be in .gitignore (TYPO3 extensions should not commit it)"
# === ADDITIONAL TEST INFRASTRUCTURE ===
- id: TT-103
type: json_path
target: composer.json
pattern: '.scripts["ci:test:php:functional"]'
severity: warning
desc: "composer.json should define ci:test:php:functional script for running functional tests via composer"
- id: TT-104
type: json_path
target: composer.json
pattern: '.scripts["ci:test:php:mutation"]'
severity: info
desc: "composer.json should define ci:test:php:mutation script for running mutation tests via composer"
# === MUTATION TESTING QUALITY ===
# `||`-chained and relying on an `expected:` field the runner never reads, so it
# was rejected unread and failed even where infection.json5 sets a high minMsi.
# Not applicable when the project does not do mutation testing at all — TT-104
# already reports the missing configuration.
#
# PCRE-free extraction for the same reason as TT-35: with `grep -oP …\K` on a
# grep without PCRE the capture is empty and the check reported "declares no
# minMsi" against a config that declares one.
#
# Reads minCoveredMsi as well as minMsi and judges the higher of the two. Since
# Infection 0.31 uncovered code is not mutated without --with-uncovered, so MSI
# equals Covered MSI and the higher threshold is the one that gates. The skill
# recommends minCoveredMsi alone (references/mutation-testing.md), which this
# check used to reject as "declares no minMsi".
#
# json5_num reads a key the way Infection does: `//` and `/* */` comments are
# dropped first (outside strings, so the `https://` of `$schema` survives), so
# a commented-out `// "minCoveredMsi": 95` is not taken for the active value.
# Keys may be double-quoted, single-quoted or bare, values may be decimal
# (both JSON5). A string value such as `"90"` is not a number and reads as
# unset. Plain POSIX awk; TT-106 carries the same function.
- id: TT-105
type: script
command: |
[ -f infection.json5 ] || exit 0
json5_num() {
awk -v key="$1" '
{ text = text $0 "\n" }
END {
n = length(text); i = 1; q = ""; out = ""
while (i <= n) {
c = substr(text, i, 1)
if (q != "") {
out = out c
if (c == "\\") { out = out substr(text, i + 1, 1); i += 2; continue }
if (c == q) q = ""
i++; continue
}
d = substr(text, i, 2)
if (d == "//") { j = index(substr(text, i), "\n"); if (j == 0) break; i += j - 1; continue }
if (d == "/*") { j = index(substr(text, i + 2), "*/"); if (j == 0) break; out = out " "; i += j + 3; continue }
if (c == "\"" || c == "\047") q = c
out = out c; i++
}
re = "(^|[^A-Za-z0-9_$])[\"\047]?" key "[\"\047]?[ \t\r\n]*:[ \t\r\n]*[0-9]+([.][0-9]+)?"
if (match(out, re)) {
v = substr(out, RSTART, RLENGTH)
sub(/^.*:[ \t\r\n]*/, "", v)
print v
}
}' "$2" 2>/dev/null
}
msi=$(json5_num minMsi infection.json5)
cmsi=$(json5_num minCoveredMsi infection.json5)
if [ -z "$msi" ] && [ -z "$cmsi" ]; then
echo "FAIL: infection.json5 declares neither minCoveredMsi nor minMsi"
exit 1
fi
gate=$(awk -v a="$msi" -v b="$cmsi" 'BEGIN { if (a == "" || (b != "" && b + 0 >= a + 0)) print b; else print a }')
awk -v g="$gate" 'BEGIN { exit !(g + 0 >= 90) }' && exit 0
echo "FAIL: infection MSI threshold is $gate (minCoveredMsi=${cmsi:-unset}, minMsi=${msi:-unset}); expected >= 90"
exit 1
severity: warning
desc: "Infection minCoveredMsi (or minMsi) should be >= 90% for mature test suites"
# Printed "consistent" or "mismatch: …" and relied on an `expected:` field the
# runner never reads; the runner judges a script by its exit status, and the
# final `echo` made that 0 either way, so a mismatch passed. It now exits 1 on
# a mismatch. Only a threshold declared in BOTH files can disagree; a missing
# file or a file without the key is TT-104's and TT-105's concern. Compares
# minMsi and minCoveredMsi separately, because a config that follows the
# skill declares only minCoveredMsi and would otherwise never be compared.
#
# PCRE-free extraction for the same reason as TT-35 and TT-105, with the same
# json5_num as TT-105: comments are ignored, whitespace before the colon
# (`"minMsi" : 90`), single-quoted and bare keys are read. Values compare as
# numbers, so 80 and 80.0 agree and 80 and 80.5 do not.
- id: TT-106
type: script
command: |
[ -f infection.json5 ] && [ -f infection-full.json5 ] || exit 0
json5_num() {
awk -v key="$1" '
{ text = text $0 "\n" }
END {
n = length(text); i = 1; q = ""; out = ""
while (i <= n) {
c = substr(text, i, 1)
if (q != "") {
out = out c
if (c == "\\") { out = out substr(text, i + 1, 1); i += 2; continue }
if (c == q) q = ""
i++; continue
}
d = substr(text, i, 2)
if (d == "//") { j = index(substr(text, i), "\n"); if (j == 0) break; i += j - 1; continue }
if (d == "/*") { j = index(substr(text, i + 2), "*/"); if (j == 0) break; out = out " "; i += j + 3; continue }
if (c == "\"" || c == "\047") q = c
out = out c; i++
}
re = "(^|[^A-Za-z0-9_$])[\"\047]?" key "[\"\047]?[ \t\r\n]*:[ \t\r\n]*[0-9]+([.][0-9]+)?"
if (match(out, re)) {
v = substr(out, RSTART, RLENGTH)
sub(/^.*:[ \t\r\n]*/, "", v)
print v
}
}' "$2" 2>/dev/null
}
rc=0
for key in minMsi minCoveredMsi; do
v1=$(json5_num "$key" infection.json5)
v2=$(json5_num "$key" infection-full.json5)
[ -z "$v1" ] || [ -z "$v2" ] && continue
awk -v a="$v1" -v b="$v2" 'BEGIN { exit !(a + 0 == b + 0) }' && continue
echo "FAIL: $key is $v1 in infection.json5 and $v2 in infection-full.json5"
rc=1
done
exit "$rc"
severity: warning
desc: "All infection config files must use consistent MSI thresholds"
# === TEST QUALITY PATTERNS ===
- id: TT-110
type: regex_not
target: Tests/**/*Test.php
pattern: 'assertEquals\s*\(\s*null\s*,'
severity: warning
desc: "Use assertNull() instead of assertEquals(null, ...) for clearer intent and better failure messages"
- id: TT-111
type: file_exists
target: Tests/Functional/Fixtures/
severity: info
desc: "Tests/Functional/Fixtures/ directory should exist for CSV fixture data used by functional tests"
# === TEST ANTI-PATTERNS ===
- id: TT-130
type: regex_not
target: Tests/**/*Test.php
pattern: "markTestSkipped\\(['\"].*(?:GD|gd|Imagick|imagick).*extension"
severity: error
desc: "Tests must not skip for missing image extensions — run tests in Docker via runTests.sh where GD/Imagick are available"
- id: TT-131
type: regex_not
target: Tests/**/*Test.php
pattern: 'createMock\(.*ImageManager::class\)'
severity: warning
desc: "Do not mock final classes (e.g., Intervention ImageManager) — use real instances or interface mocks"
# === TEST COVERAGE COMPLETENESS ===
# `-not -path` excludes a dependency tree vendored underneath the search
# root. The exclusions are spelled `*/vendor/*` rather than `./vendor/*`:
# find starts at Classes/Service, so it emits paths beginning `Classes/`,
# and a `./`-anchored pattern could never match one.
#
# The counter used to be incremented inside `find … | while …`, which bash
# runs in a subshell: `missing` came back 0 no matter how many classes were
# reported, so the check printed its findings and still exited 0 — it could
# not fail. Process substitution keeps the loop in the current shell.
#
# Spelled `type: script` + `command: |` — the only multi-line shape both the
# runner and validate-checkpoints.sh accept (see TD-05 in typo3-docs).
#
# Only files that declare behaviour need a test of their own: interfaces,
# abstract classes (tested through their concrete subclasses) and classes or
# enums without a single method (constants holders, backed enums) are
# skipped. The PHP tokenizer decides, so a docblock mentioning `interface`
# does not count, and neither does a `use function` import.
- id: TT-120
type: script
command: |
missing=0
while IFS= read -r f; do
base=$(basename "$f" .php)
rel=${f#Classes/}
dir=$(dirname "$rel")
if [ ! -f "Tests/Unit/${dir}/${base}Test.php" ] && [ ! -f "Tests/Functional/${dir}/${base}Test.php" ]; then
echo "Missing test for $f"
missing=$((missing+1))
fi
done < <(find Classes/Service -name '*.php' -not -path '*/.Build/*' -not -path '*/vendor/*' -not -path '*/node_modules/*' 2>/dev/null | php -r '
foreach (array_filter(explode("\n", stream_get_contents(STDIN))) as $f) {
$t = array_values(array_filter(token_get_all(file_get_contents($f)), fn($x) => !(is_array($x) && in_array($x[0], [T_WHITESPACE, T_COMMENT, T_DOC_COMMENT], true))));
$kind = null; $abstract = false; $methods = 0;
foreach ($t as $i => $x) {
if (!is_array($x)) continue;
if ($kind === null && in_array($x[0], [T_CLASS, T_INTERFACE, T_TRAIT, T_ENUM], true) && !(is_array($t[$i - 1] ?? null) && $t[$i - 1][0] === T_DOUBLE_COLON)) {
$kind = $x[0];
for ($j = $i - 1; $j >= 0 && is_array($t[$j]) && in_array($t[$j][0], [T_ABSTRACT, T_FINAL, T_READONLY], true); $j--) {
if ($t[$j][0] === T_ABSTRACT) $abstract = true;
}
}
if ($x[0] === T_FUNCTION && !(is_array($t[$i - 1] ?? null) && $t[$i - 1][0] === T_USE)) $methods++; // `use function foo;` is an import, not a method
}
if ($kind === null || $kind === T_INTERFACE || $abstract || $methods === 0) continue;
echo $f, "\n";
}')
[ "$missing" -eq 0 ]
severity: warning
desc: "Every class or enum with methods in Classes/Service/ (including subdirectories) should have a corresponding test; interfaces and abstract classes are exempt"
# Abstract ViewHelper bases are exempt: they are tested through their
# concrete subclasses. Interfaces, traits and enums are skipped too, since
# none of them is a ViewHelper. Unlike TT-120 a concrete class without own
# methods is NOT
# exempt: `final class FooViewHelper extends AbstractFooViewHelper {}`
# inherits render() and is a ViewHelper that templates call, so it needs a test.
- id: TT-121
type: script
command: |
missing=0
while IFS= read -r f; do
base=$(basename "$f" .php)
rel=${f#Classes/}
dir=$(dirname "$rel")
if [ ! -f "Tests/Unit/${dir}/${base}Test.php" ] && [ ! -f "Tests/Functional/${dir}/${base}Test.php" ]; then
echo "Missing test for $f"
missing=$((missing+1))
fi
done < <(find Classes/ViewHelpers -name '*ViewHelper.php' -not -path '*/.Build/*' -not -path '*/vendor/*' -not -path '*/node_modules/*' 2>/dev/null | php -r '
foreach (array_filter(explode("\n", stream_get_contents(STDIN))) as $f) {
$t = array_values(array_filter(token_get_all(file_get_contents($f)), fn($x) => !(is_array($x) && in_array($x[0], [T_WHITESPACE, T_COMMENT, T_DOC_COMMENT], true))));
$kind = null; $abstract = false;
foreach ($t as $i => $x) {
if (!is_array($x)) continue;
if ($kind === null && in_array($x[0], [T_CLASS, T_INTERFACE, T_TRAIT, T_ENUM], true) && !(is_array($t[$i - 1] ?? null) && $t[$i - 1][0] === T_DOUBLE_COLON)) {
$kind = $x[0];
for ($j = $i - 1; $j >= 0 && is_array($t[$j]) && in_array($t[$j][0], [T_ABSTRACT, T_FINAL, T_READONLY], true); $j--) {
if ($t[$j][0] === T_ABSTRACT) $abstract = true;
}
}
}
if ($kind !== T_CLASS || $abstract) continue;
echo $f, "\n";
}')
[ "$missing" -eq 0 ]
severity: warning
desc: "Every concrete ViewHelper in Classes/ViewHelpers/ (including subdirectories) should have a corresponding test; abstract bases are exempt"
# === ADDITIONAL CHECKPOINTS (t3x-nr-vault v0.5.0 session) ===
- id: TT-132
type: regex_not
target: Tests/E2E/**/*.spec.ts
pattern: "(['\"])\\s*/(?:home|Users)/[^'\"/]+/[^'\"]*\\1|(['\"])C:\\\\\\\\"
severity: error
desc: "E2E specs must not contain hardcoded absolute local paths (/home/username/..., /Users/username/..., C:\\...) — breaks CI and every other developer's machine. Use process.cwd() or paths relative to the test file instead."
note: "Real bug caught in t3x-nr-vault review: hardcoded /home/cybot/projects/.../main in an execFileSync cwd."
- id: TT-135
type: regex_not
target: Tests/Functional/**/*Test.php
pattern: 'function setUp\(\)[^}]*(?:(?:self|static)::markTestSkipped|\$this->markTestSkipped)[^}]*parent::setUp\(\)'
multiline: true
severity: error
desc: >-
markTestSkipped() must not be called before parent::setUp() in a FunctionalTestCase::setUp().
Typed properties like $this->instancePath are uninitialised until parent::setUp() runs;
tearDown() then crashes with "accessed before initialization".
Correct order: parent::setUp() first, then check the skip condition.
llm_reviews:
# === MOVED FROM mechanical: — type llm_review is not a mechanical type,
# so the runner skipped these three as 'Unknown checkpoint type' and they
# never ran anywhere. ===
- id: TT-133
type: llm_review
domain: test-quality
target: "Tests/Unit/**/*Test.php"
severity: warning
desc: "#[CoversClass] target must not appear in phpunit.xml <source><exclude>. PHPUnit 12 emits 'not a valid target for code coverage' when a class is both covered and excluded; failOnWarning=true promotes this to a failure. Use #[CoversNothing] for tests of excluded classes, or remove the class from the exclude list."
- id: TT-134
type: llm_review
domain: test-quality
target: "Tests/**/*.php"
severity: error
desc: "Tests that call vfsStream::url() must call vfsStream::setup() for the same root in setUp(). On a fresh CI run the virtual-filesystem root does not exist; file_put_contents('vfs://keys/...') fails with 'No such file or directory'. Works locally only when a previous test in the same process already registered the root."
# Same candidate paths as TT-30/TT-32/TT-35: a project whose PHPStan config
# lives in Build/ was skipped by this review entirely.
- id: TT-136
type: llm_review
domain: static-analysis
target: "{phpstan.neon,Build/phpstan.neon,Build/phpstan/phpstan.neon}"
severity: error
desc: "phpstan.neon must not combine phpstan/extension-installer auto-registration with explicit includes: of the same extension .neon files. When extension-installer is active (via netresearch/typo3-ci-workflows), phpstan-phpunit, phpstan-strict-rules, phpstan-deprecation-rules, phpat, and saschaegerer/phpstan-typo3 are already registered — adding explicit includes: causes 'These files are included multiple times' and PHPStan exits 1. Remove duplicate includes or use Build/phpstan.no-plugins.neon for local runs without extension-installer."
# === DEPENDENCY FRESHNESS ===
- id: TT-19
type: llm_review
severity: info
domain: dependencies
prompt: |
If package.json exists, check that test framework dependencies
(vitest, playwright, jest, jsdom) are not more than 2 major
versions behind the latest release. Outdated test tools can
cause false positives/negatives and miss new testing features.
desc: "npm test dependencies not excessively outdated"
tags: [npm, dependencies, testing]
# === TEST QUALITY REVIEWS ===
- id: TT-85
domain: code-quality
prompt: |
Review test files for incorrect #[CoversClass] attribute usage:
- Each #[CoversClass(Foo::class)] must reference a class that the test
actually exercises (creates, calls methods on, or asserts behavior of).
- Flag any test that declares #[CoversClass] on a class it does not
instantiate, mock, or invoke methods on within any test method.
- Common mistake: copy-pasting CoversClass from another test file.
Examine Tests/Unit/**/*Test.php and Tests/Functional/**/*Test.php.
severity: warning
desc: "Verify #[CoversClass] attributes reference classes the test actually exercises"
- id: TT-80
domain: code-quality
prompt: |
Review the test directory structure and naming conventions:
- Tests should be in Tests/Unit/ and Tests/Functional/
- Test class names should end with "Test"
- Test methods should start with "test" or use #[Test] attribute
- Namespace should mirror Classes/ structure
Check Tests/ directory and report compliance.
severity: warning
desc: "Verify test structure follows TYPO3 conventions"
- id: TT-81
domain: code-quality
prompt: |
Review PHPUnit configuration for best practices:
- Bootstrap file should be configured
- Test suites should be properly defined
- Code coverage filter should exclude Tests/ and vendor/
- Fail on warnings/risky should be enabled
Examine phpunit.xml, Build/UnitTests.xml, Build/phpunit.xml or
Build/phpunit/UnitTests.xml and report findings.
severity: info
desc: "Verify PHPUnit configuration follows best practices"
- id: TT-82
domain: code-quality
prompt: |
Review CI test matrix configuration:
- Should test against supported PHP versions (8.2, 8.3, 8.4, 8.5)
- Should test against supported TYPO3 versions (v13, v14)
- If using netresearch/typo3-ci-workflows, check that php-versions
and typo3-versions inputs are properly configured
- Should have reasonable timeout limits
Examine .github/workflows/*.yml and report matrix coverage.
severity: warning
desc: "Verify CI matrix covers required PHP and TYPO3 versions"
- id: TT-83
domain: code-quality
prompt: |
Review PHPStan configuration for TYPO3 extension development:
- Should include saschaegerer/phpstan-typo3 extension
- Level should be 6 or higher (ideally 9/max, or 10 with PHPStan 2.x)
- Should scan Classes/ directory
- Baseline file is acceptable for legacy code
Examine phpstan.neon (root or Build/ directory) and report quality.
severity: info
desc: "Verify PHPStan is properly configured for TYPO3"
- id: TT-84
domain: code-quality
prompt: |
Review test coverage and quality indicators:
- Are there tests for core functionality?
- Do tests cover edge cases and error conditions?
- Are mocks used appropriately?
- Is test isolation maintained?
Sample Tests/Unit/ and Tests/Functional/ directories and assess quality.
severity: info
desc: "Assess overall test coverage and quality"
# === TEST INFRASTRUCTURE ANTI-PATTERNS ===
- id: TT-91
domain: test-quality
prompt: |
Review test code for markTestSkipped guards that hide missing Docker infrastructure:
1. Find all uses of markTestSkipped() or $this->markTestSkipped() in test files
2. Check if any skip guards test for PHP extensions (GD, Imagick, etc.) or
external tools that should be available in the Docker test environment
3. Tests for image/file/network operations should always run in containerized
environments via runTests.sh, not be skipped on bare-metal
4. Flag any markTestSkipped() that tests for extensions or binaries that the
Docker container should provide
severity: warning
desc: "Review test code for markTestSkipped guards that hide missing Docker infrastructure — tests for image/file/network operations should always run in containerized environments"
- id: TT-92
domain: test-quality
prompt: |
Review test code for mocking of final or readonly classes:
1. Find all createMock() and getMockBuilder() calls in test files
2. Resolve the target class and check if it is declared final or readonly
(e.g., Intervention\Image\ImageManager is final in v3+)
3. Mocking a final class causes a runtime error in PHPUnit
4. Suggest alternatives: use real instances, mock interfaces instead,
or use adapter patterns that wrap the final class
5. Flag any createMock(FinalClass::class) usage
severity: warning
desc: "Review for mocking of final or readonly classes — use real instances, interface mocks, or adapter patterns instead of createMock() on final classes"
# === TEST COVERAGE COMPLETENESS ===
- id: TT-87
domain: code-quality
prompt: |
Check every class in Classes/Service/ has a corresponding test in Tests/Unit/Service/ or Tests/Functional/Service/:
1. List all PHP classes in Classes/Service/
2. For each class, check if a matching test file exists in Tests/Unit/Service/ or Tests/Functional/Service/
3. Flag any service class without a corresponding test
severity: warning
desc: "Every service class in Classes/Service/ should have a test in Tests/Unit/Service/ or Tests/Functional/Service/"
- id: TT-88
domain: code-quality
prompt: |
Check every ViewHelper in Classes/ViewHelpers/ has a corresponding test:
1. List all ViewHelper classes in Classes/ViewHelpers/ (recursively)
2. For each ViewHelper, check if a matching test file exists in Tests/Unit/ViewHelpers/ or Tests/Functional/ViewHelpers/
3. Flag any ViewHelper without a corresponding test
severity: warning
desc: "Every ViewHelper in Classes/ViewHelpers/ should have a corresponding test"
# === EVENT DISPATCH AND ERROR HANDLING ===
- id: TT-77
domain: testing
prompt: |
Review test coverage for event dispatch error handling:
1. Find all try/catch blocks that wrap EventDispatcherInterface::dispatch() calls
2. Verify each catch block has a dedicated test that triggers the exception path
3. Check that tests mock the EventDispatcher to throw exceptions and verify the catch behavior
4. Flag any event dispatch catch block without corresponding test coverage
severity: warning
desc: "Event dispatch catch blocks must have dedicated tests that verify exception handling behavior"
- id: TT-78
domain: testing
prompt: |
Review test coverage for PHP error suppression and trigger_error:
1. Find all uses of @ error suppression operator in production code
2. Verify each @-suppressed call has tests covering both success and failure paths
3. Find functions that are known to trigger PHP warnings on failure (e.g., getimagesize, file_get_contents, mkdir, unlink, fopen)
4. Verify tests exist that trigger the warning path (e.g., invalid file input)
5. Check that tests don't just suppress warnings but explicitly test error handling
6. Find all trigger_error() calls in production code and the code paths that rely on them
7. Verify tests assert how each triggered error is handled or propagated (e.g., custom error handlers, exceptions, logging)
severity: warning
desc: "Functions with @ suppression, trigger_error(), or that trigger PHP warnings must have tests covering error paths"
# === CONTROLLER FUNCTIONAL TEST COVERAGE ===
- id: TT-86
domain: testing
prompt: |
Review backend controller test coverage:
1. Find all classes in Classes/Controller/ (especially AJAX controllers returning JsonResponse)
2. For each controller, check that functional tests exist in Tests/Functional/Controller/
3. Verify that functional tests check:
a. HTTP status codes (200 for success, 403 for unauthorized, etc.)
b. Access control (calling without proper BE user session returns 403/401)
c. CSRF token validation (requests without valid token are rejected)
4. Flag any backend controller without functional tests covering these three aspects
severity: warning
desc: "Every backend controller (especially AJAX controllers returning JSON) should have functional tests verifying HTTP status codes, access control, and CSRF"
# === CI TEST MATRIX COMPLETENESS ===
- id: TT-27
domain: ci
prompt: |
Verify that the CI test matrix covers all supported version combinations:
1. Read the PHP version constraint from composer.json require.php (e.g., "^8.2")
2. Read the TYPO3 core constraint from composer.json require["typo3/cms-core"] (e.g., "^13.4 || ^14.0")
3. Determine which PHP minor versions fall in the constraint range (e.g., 8.2, 8.3, 8.4)
4. Determine which TYPO3 major versions fall in the constraint range (e.g., 13, 14)
5. Check the CI workflow (.github/workflows/*.yml) matrix or reusable workflow inputs
6. Flag any PHP version in the supported range that is missing from the CI matrix
7. Flag any TYPO3 version in the supported range that is missing from the CI matrix
8. If using netresearch/typo3-ci-workflows, check php-versions and typo3-versions inputs
severity: warning
desc: "CI should test all PHP versions in composer.json require.php range x all TYPO3 versions in require[\"typo3/cms-core\"] range"
# === E2E MULTI-BROWSER COVERAGE ===
- id: TT-17b
domain: testing
prompt: |
If a Playwright config exists (Build/playwright.config.ts or playwright.config.ts):
1. Read the projects array in the Playwright config
2. Verify it defines at least two browser projects: Chromium and Firefox
3. Optionally check for WebKit as a third browser
4. Flag if only a single browser is configured, as cross-browser testing catches
rendering and API differences that single-browser testing misses
severity: warning
desc: "If playwright.config.ts exists, it should define at least Chromium and Firefox projects for cross-browser E2E coverage"
# === PATCH COVERAGE ===
- id: TT-89
domain: testing
prompt: |
Review whether new code paths introduced in recent changes have corresponding tests:
1. Identify new public methods in Classes/ that were added or significantly changed
2. For each new public method, check whether a test exists that calls or exercises it
3. Focus on Classes/Service/, Classes/Controller/, and Classes/ViewHelpers/ directories
4. Flag any new public method that has no corresponding test coverage
5. Ignore getters/setters, constructors, and simple delegation methods
severity: warning
desc: "New code paths (especially new public methods) should have corresponding unit or functional tests"
# === TEST UID SETUP ===
- id: TT-112
domain: testing
prompt: |
Review test helpers and factories that create entity or model mocks:
1. Find helper methods that create domain model instances for testing
(e.g., createSecretEntity, createMockRecord, buildFixture)
2. For each helper, check whether the production code that consumes
these entities guards on UID being set (e.g., `if ($uid !== null)`,
`if ($entity->getUid() > 0)`, `$uid ?? throw`)
3. If the production code has a UID null-check, verify the test helper
sets a UID via `_setProperty('uid', ...)` or `setUid(...)` or reflection
4. Flag any test helper that creates entities without setting UID when
the consuming production code guards on UID presence
severity: warning
desc: "Test helpers creating entity mocks must set UID when production code guards with null/zero UID checks"
# === MOCK CONSTRUCTOR SYNC ===
- id: TT-113
domain: testing
prompt: |
Review whether test mocks stay in sync with class constructors:
1. Find classes in Classes/ whose constructors have changed recently
or have 3+ parameters (likely to drift)
2. For each class, find all tests that instantiate it directly
(new ClassName(...)) or via createMock/getMockBuilder
3. Check that direct instantiations pass the correct number and types
of constructor arguments
4. For createMock/getMockBuilder, check if disableOriginalConstructor()
is used — if not, the mock will call the real constructor and must
match its signature
5. Flag any test where constructor argument count or types are mismatched
with the current class constructor
severity: warning
desc: "When a class constructor gains new parameters, all tests that instantiate it must be updated to match"
# === MOCK VALIDITY FOR MULTI-VERSION DEPENDENCIES ===
- id: TT-114
domain: testing
prompt: |
Review test mocks for multi-version dependency compatibility:
1. Read composer.json require and require-dev for dependencies with
multi-major-version constraints (e.g., "^3 || ^4", "^2 || ^3")
2. For each such dependency, find all test files that mock its interfaces
(grep for createMock/createStub with the dependency's namespace)
3. For each mocked method, verify the method exists on the **interface**
(not just the concrete class) in ALL supported major versions
4. Flag any mock that calls ->method('foo') where foo only exists on
a concrete implementation or only in one major version
5. Suggest using an adapter interface pattern when version-specific
mocking is detected
severity: warning
desc: "Test mocks on dependency interfaces must reference methods that exist in ALL supported versions of the dependency"
# === MOCK CALLBACK SIGNATURE VERIFICATION ===
- id: TT-115
domain: testing
prompt: |
Review willReturnCallback() usages for signature mismatches:
1. Find all willReturnCallback() calls in test files
2. For each callback, identify the method being mocked
3. Compare the callback's parameter list with the actual method signature
in the production class or interface
4. Flag callbacks whose parameter count or types do not match the
actual method signature
5. Flag callbacks that are missing parameters that the production code
now passes (e.g., production added a $quality parameter but the
callback only accepts $path)
6. Suggest using variadic signatures (mixed ...$options) when the
method signature may evolve across dependency versions
severity: warning
desc: "willReturnCallback() signatures must match the actual method signature; stale callbacks silently drop arguments"
# === TEST ASSERTION SPECIFICITY AFTER REFACTORING ===
- id: TT-116
domain: testing
prompt: |
Review whether test assertions maintain equivalent specificity after
production code refactoring:
1. Find test methods that assert on method calls (expects/method/with)
2. Compare these assertions with the current production code paths
3. Flag tests where a refactoring changed the production API surface
but the test assertion lost specificity. Examples:
- Old: expects(never())->method('toWebp') → New: expects(once())->method('save')
(lost the "not WebP" assertion)
- Old: expects(once())->method('resize')->with(800, 600) → New: just expects(once())->method('process')
(lost dimension verification)
4. Verify that expects(exactly(N)) with callback assertions include
argument-level checks (e.g., asserting path extensions, dimensions)
5. Flag generic willReturnCallback() that only provides return values
without asserting anything about the arguments
severity: warning
desc: "After refactoring, test assertions must maintain equivalent specificity — don't trade specific method assertions for generic ones"
# === ADAPTER PATTERN TESTING ===
- id: TT-117
domain: testing
prompt: |
Review whether tests mock third-party library internals directly
when an adapter pattern would be more appropriate:
1. Find test files that create mocks of third-party library interfaces
or classes (namespaces outside the extension's own namespace)
2. Check if the mocked third-party class has version-specific APIs
(compare composer.json constraints for multi-major-version support)
3. Flag tests that create complex mock setups for third-party internals
(e.g., mocking DriverInterface + constructing real ImageManager)
4. Suggest introducing an adapter interface owned by the extension
and mocking that instead, which eliminates version-specific mock setup
5. Exception: simple, stable interfaces (like PSR interfaces) are fine
to mock directly
severity: warning
desc: "Prefer mocking your own adapter interfaces over third-party library internals to avoid version-specific mock breakage"
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |