netresearch/typo3-testing-skill / skills/typo3-testing
typo3-testing
FlaggedUse when reproducing defects as failing tests, testing template or rendered output changes, confirming changes on a running TYPO3 instance, or setting up TYPO3 extension tests: unit/functional/E2E, PHPUnit 11-13, mutation testing, final-class mocks (v14), CI matrix across TYPO3 12/13/14.3 LTS, typo3-ci-workflows meta-package, CI failures. Also: transaction rollback, worker crashes, competing requests, testing-framework setup, test matrix, integration/e2e testing, coverage, test generation.
references/ci-cd.md
View on GitHub<!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
<!-- SPDX-FileCopyrightText: Netresearch DTT GmbH -->
# CI/CD Integration for TYPO3 Testing
Continuous Integration and Continuous Deployment workflows for automated TYPO3 extension testing.
## GitHub Actions
### Basic Workflow
Create `.github/workflows/tests.yml`:
```yaml
name: Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
jobs:
lint:
name: Lint PHP
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.4'
- name: Install dependencies
run: composer install --no-progress
- name: Run linting
run: composer ci:test:php:lint
phpstan:
name: PHPStan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.4'
- name: Install dependencies
run: composer install --no-progress
- name: Run PHPStan
run: composer ci:test:php:phpstan
unit:
name: Unit Tests
runs-on: ubuntu-latest
strategy:
matrix:
php: ['8.1', '8.2', '8.3', '8.4']
steps:
- uses: actions/checkout@v4
- name: Setup PHP ${{ matrix.php }}
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
coverage: xdebug
- name: Install dependencies
run: composer install --no-progress
- name: Run unit tests
run: composer ci:test:php:unit
- name: Upload coverage
# Upload coverage for all PHP versions
uses: codecov/codecov-action@v3
functional:
name: Functional Tests
runs-on: ubuntu-latest
strategy:
matrix:
php: ['8.1', '8.2', '8.3', '8.4']
database: ['mysqli', 'pdo_mysql', 'postgres', 'sqlite']
services:
mysql:
image: mysql:8.0
env:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: typo3_test
ports:
- 3306:3306
options: >-
--health-cmd="mysqladmin ping"
--health-interval=10s
--health-timeout=5s
--health-retries=3
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: typo3_test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- name: Setup PHP ${{ matrix.php }}
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
extensions: ${{ matrix.database == 'postgres' && 'pdo_pgsql' || 'mysqli' }}
- name: Install dependencies
run: composer install --no-progress
- name: Run functional tests
run: |
export typo3DatabaseDriver=${{ matrix.database }}
export typo3DatabaseHost=127.0.0.1
export typo3DatabaseName=typo3_test
export typo3DatabaseUsername=${{ matrix.database == 'postgres' && 'postgres' || 'root' }}
export typo3DatabasePassword=${{ matrix.database == 'postgres' && 'postgres' || 'root' }}
composer ci:test:php:functional
```
### Matrix Strategy
Test multiple PHP and TYPO3 versions:
```yaml
strategy:
fail-fast: false
matrix:
php: ['8.1', '8.2', '8.3', '8.4']
typo3: ['12.4', '13.0']
exclude:
- php: '8.1'
typo3: '13.0' # TYPO3 v13 requires PHP 8.2+
steps:
- name: Install TYPO3 v${{ matrix.typo3 }}
run: |
composer require "typo3/cms-core:^${{ matrix.typo3 }}" --no-update
composer update --no-progress
```
### Adding PHP 8.4/8.5 breaks a `php -l` lint step that greps stdout
Where a project lints by grepping `php -l` output instead of using a linter with
its own exit codes, the step fails the moment the matrix reaches PHP 8.4 — for a
reason that has nothing to do with the project's code. The scaffolding in this
skill uses `phplint` (`ci:test:php:lint`) and is unaffected; this is about a
hand-rolled step of the following shape, seen in
`TYPO3-Documentation/guides-php-domain`:
```yaml
# Breaks on PHP >= 8.4 once dependencies are installed
run: find . -type f -name '*.php' ! -path "./.Build/*" -print0
| xargs -0 -n1 -P4 php -l -n
| (! grep -v "No syntax errors detected" )
```
PHP 8.4 deprecated implicitly nullable parameter types, and `php -l` writes that
deprecation to **stdout**, not stderr. The `grep -v` therefore sees a line that
is not `No syntax errors detected` and the step fails — reporting a syntax error
in a file whose syntax is fine. Because the notice comes from installed
dependencies, the failure appears only after `composer install` and only on the
new matrix legs, which makes it read like a genuine incompatibility in the
dependency tree.
`php -l` is not the thing that disagrees. On one such file at PHP 8.5.6:
```console
$ php -l -n vendor/…/FilesystemInterface.php 2>/dev/null # stdout alone
Deprecated: League\Flysystem\FilesystemInterface::get(): Implicitly marking parameter $handler as nullable is deprecated, … on line 286
No syntax errors detected in vendor/…/FilesystemInterface.php
$ php -l -n vendor/…/FilesystemInterface.php 2>&1 1>/dev/null # stderr alone
# (empty)
$ php -l -n vendor/…/FilesystemInterface.php >/dev/null 2>&1; echo $?
0
```
Both lines are on stdout, stderr is empty, and the linter's own exit code is 0 —
the failure is manufactured entirely by the `grep` wrapper. Redirecting stderr
therefore changes nothing.
Measured on `TYPO3-Documentation/guides-php-domain` at PHP 8.5.6: the command
above emits **478** such lines and exits 1, from `phpdocumentor/filesystem`,
`localheinz/diff` and `rector/rector`, among others.
Scope the lint to your own sources — third-party code is not yours to
syntax-check:
```yaml
run: find . -type f -name '*.php' ! -path "./.Build/*" ! -path "./vendor/*" -print0
| xargs -0 -n1 -P4 php -l -n
| (! grep -v "No syntax errors detected" )
```
Two tempting non-fixes. **Dropping `-n`** does make the message go away — `-n`
suppresses `php.ini`, so without it an ini carrying
`error_reporting = E_ALL & ~E_DEPRECATED` or `display_errors = Off` hides the
notice and the step goes green (measured: both leave only
`No syntax errors detected`). That is worse than the bug, because the lint's
result now depends on ambient ini configuration that differs between the runner
and every developer machine, and the same suppression hides deprecations in
*your* code. **Filtering the deprecation text** out of the `grep` hides real
diagnostics that arrive on the same channel. Scoping the path is the fix that
narrows *what* is linted rather than *what is reported*.
### Reproducing one matrix leg locally before pushing
A matrix change is worth verifying per leg, and the official images make that a
single command — no local PHP juggling, no DDEV:
```bash
mkdir -p /tmp/leg85 # tar -x -C exits 2 if the destination does not exist
tar -c --exclude=vendor --exclude=.Build --exclude=.git . | tar -x -C /tmp/leg85
docker run --rm -v /tmp/leg85:/app -w /app php:8.5-cli sh -c '
curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer --quiet
composer install --no-interaction --no-progress
composer show symfony/console # WHICH dependency set did this leg resolve?
vendor/bin/phpunit --testsuite=unit'
```
Copy the working tree with `tar`; `git archive` drops every `export-ignore`d
path, which is typically `tests/` and the analyser configs.
**`composer show <dep>` is the step that earns the run.** Where no
`composer.lock` is committed, each leg resolves fresh and different PHP versions
get different major versions of the same dependency — so "does the suite pass on
8.5?" and "what does 8.5 install?" are two questions, and only the second
explains a failure. Same repo, same branch, one command apart:
| | PHP 8.2.30 | PHP 8.5.6 |
|---|---|---|
| `symfony/console` resolved | v7.4.19 | v8.1.7 |
That split is what decides whether a static-analysis job is affected. On the 8.5
leg, PHPStan 1.12.34 against `symfony/console` v8.1.7 exits 1 with two false
`Call to an undefined method …Command\Run::getDefinition()` errors — it cannot
resolve the Symfony 8 base class, so it invents findings in code that is fine.
The analysis job pins a single PHP version, though, and on 8.2 the resolve
yields Symfony 7, where the same run is clean. Without the two numbers that is
an argument; with them it is a measurement, and no dependency cap is needed.
### Caching Dependencies
```yaml
- name: Cache Composer dependencies
uses: actions/cache@v3
with:
path: ~/.composer/cache
key: composer-${{ runner.os }}-${{ matrix.php }}-${{ hashFiles('composer.lock') }}
restore-keys: |
composer-${{ runner.os }}-${{ matrix.php }}-
composer-${{ runner.os }}-
- name: Install dependencies
run: composer install --no-progress --prefer-dist
```
### Code Coverage
```yaml
- name: Run tests with coverage
run: vendor/bin/phpunit -c Build/phpunit/UnitTests.xml --coverage-clover coverage.xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
flags: unittests
name: codecov-umbrella
```
### Xdebug vs PCOV for Coverage
**Xdebug is recommended for CI/CD.** PCOV is faster but gives up enough diagnostic fidelity and local/CI parity that the tradeoff rarely pays off in practice.
| Aspect | Xdebug | PCOV |
|--------|--------|------|
| **Local/CI parity** | ✅ Local coverage runs typically set `XDEBUG_MODE=coverage`; keeping CI on xdebug means local and CI behave identically | ❌ Mismatches local; leaks `beStrictAboutCoverageMetadata`-style drift |
| **Branch coverage** | ⚠️ Only when requested: `--path-coverage` (branches and paths), or `--branch-coverage` (branches only, PHPUnit 13.2+); line-only without either | ❌ Line-only |
| **Purpose** | Debugger + Profiler + Coverage | Coverage only |
| **Speed** | Slower (debugger overhead) | 2-5× faster |
| **Memory** | Higher (full debugger loaded) | Lower footprint |
**Why Xdebug is the better default:**
- **Strict coverage metadata**: PHPUnit's `beStrictAboutCoverageMetadata="true"` marks tests as "risky" when they execute code outside their declared `#[CoversClass]` / `#[UsesClass]` attributes. The check only runs under active coverage. Mixing local Xdebug with CI PCOV produced "green locally, red in CI" surprises — switching both to Xdebug eliminates that drift.
- **Branch + path coverage is opt-in**: Xdebug can see `if/else` branches and early returns, but only when PHPUnit runs with `--path-coverage`, or, for branches without paths, `--branch-coverage` / `branchCoverage="true"` (PHPUnit 13.2 and later; 11.5, 12.5 and 13.1 do not have it). Without either flag the clover totals show `conditionals="0"`, identical to PCOV. Whether `--branch-coverage` avoids the Xdebug 3.5 crash below is not tested. With the flag, the branch signal helps when preparing mutation testing; see "Branch and path coverage" below before enabling it.
- **Cost**: ~2-3 min extra CI runtime across a typical 8-job matrix. For a sharded functional suite, measured on GitHub runners, one cell took 992–1392 s with Xdebug line coverage, 364–818 s with PCOV and 197–381 s without coverage.
**When PCOV is still the right call:**
- CI matrix has so many jobs (e.g. 20+ combinations) that the 2-5× coverage speed-up genuinely matters.
- Coverage runs are gated (only on the default branch, not every PR).
- You're comfortable maintaining `[CoversClass]` / `[UsesClass]` declarations that pass locally and in CI — see the "Strict coverage metadata" note below.
**Via `netresearch/typo3-ci-workflows`:**
As of [netresearch/typo3-ci-workflows#72](https://github.com/netresearch/typo3-ci-workflows/pull/72), the default `coverage-tool` is `xdebug`. Consumers can still override explicitly:
```yaml
jobs:
ci:
uses: netresearch/typo3-ci-workflows/.github/workflows/ci.yml@main
with:
upload-coverage: true
# coverage-tool defaults to xdebug (recommended)
# coverage-tool: 'pcov' # opt in to pcov for faster runs
```
**Standalone GitHub Actions setup:**
```yaml
- name: Setup PHP with Xdebug (recommended)
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
coverage: xdebug
# Or opt into PCOV when CI runtime matters more than coverage fidelity:
- name: Setup PHP with PCOV
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
coverage: pcov
```
**Local execution via `Build/Scripts/runTests.sh`:**
The canonical TYPO3 core-testing pattern runs the suite inside the
`ghcr.io/typo3/core-testing-*` Docker images, which already have
xdebug available. Pass `XDEBUG_MODE=coverage` into the container so
PHPUnit picks xdebug (not pcov, when both are installed):
```bash
# Recommended: xdebug coverage via runTests.sh
XDEBUG_MODE=coverage Build/Scripts/runTests.sh -s unit -- \
--coverage-clover=coverage.xml
# Functional suite with coverage
XDEBUG_MODE=coverage Build/Scripts/runTests.sh -s functional -- \
--coverage-clover=coverage-functional.xml
```
**Direct PHPUnit (when runTests.sh is not in use):**
```bash
# xdebug (matches CI default)
php -d xdebug.mode=coverage vendor/bin/phpunit \
--coverage-clover coverage.xml
# pcov (opt-in, speed over fidelity)
php -d pcov.enabled=1 -d xdebug.mode=off vendor/bin/phpunit \
--coverage-clover coverage.xml
```
#### Branch and path coverage
- `netresearch/typo3-ci-workflows` has the input `path-coverage` (added in [netresearch/typo3-ci-workflows#312](https://github.com/netresearch/typo3-ci-workflows/pull/312)), default `false`. It adds `--path-coverage` to the built-in unit, functional and acceptance runs when `upload-coverage` is on and `coverage-tool` is `xdebug`. A custom `*-test-command` receives `PHPUNIT_PATH_COVERAGE=true` in its environment and must pass the flag itself. The same PR added `unit-timeout-minutes` (15), `functional-timeout-minutes` (30) and `acceptance-timeout-minutes` (15).
- Why the default is off: with `--path-coverage`, Xdebug 3.5.0, 3.5.1, 3.5.3 and the development branch (64007df, 3.6.0alpha2-dev) crash with a segmentation fault in `xdebug_branch_info_mark_reached` (`src/coverage/branch_info.c:384`) on netresearch/t3x-nr-llm's suites. One functional test method crashed 6 of 6 runs on each of those versions and 0 of 6 on 3.4.7 (PHP 8.4.25, `ghcr.io/typo3/core-testing-php84`). Xdebug 3.4 does not support PHP 8.5. Related upstream report with the same crash site, open since 3.3.1: <https://bugs.xdebug.org/view.php?id=2252>. How to bisect such a crash: [ci-debugging.md](ci-debugging.md), "A PHP extension segfaults in functional tests".
- Codecov's per-line view stays line-based even with path coverage: php-code-coverage's Clover and Cobertura writers (checked in 11.0.12 and 14.4.1) write branch counts only in the metric totals, not per line.
```yaml
jobs:
ci:
uses: netresearch/typo3-ci-workflows/.github/workflows/ci.yml@main
with:
upload-coverage: true
coverage-tool: xdebug
path-coverage: true # opt-in; see the Xdebug 3.5 crash above
# path coverage makes runs slower: raise unit-, functional- or
# acceptance-timeout-minutes where the defaults are too short
```
#### Strict coverage metadata
If `Build/UnitTests.xml` / `Build/FunctionalTests.xml` sets `beStrictAboutCoverageMetadata="true"` together with `failOnRisky="true"`, every test must declare every class it executes via `#[CoversClass]` or `#[UsesClass]` — otherwise the coverage-driven check flags the test as risky and fails the run.
- Unit tests: strict metadata is natural — a unit test touches exactly one class.
- Functional / integration tests: expect to declare the full transitive dependency chain via `#[UsesClass]`. Example:
```php
#[CoversClass(ProcessingMiddleware::class)]
#[UsesClass(Processor::class)]
#[UsesClass(ImageManagerAdapter::class)]
#[UsesClass(ImageManagerFactory::class)]
#[UsesClass(VariantServedEvent::class)]
final class ProcessingMiddlewareTest extends FunctionalTestCase
```
If maintaining those lists across DI refactorings costs too much, relax to `beStrictAboutCoverageMetadata="false"` *on functional tests only* — keep unit tests strict.
> **Note:** PHPUnit auto-detects available coverage drivers. If both are
> present (some Docker images install both), PHPUnit prefers PCOV — set
> `XDEBUG_MODE=coverage` or pass `-d pcov.enabled=0` to force Xdebug.
### Sharded functional tests with coverage
- **Pattern**: one PHPUnit process per test class. That is safe because typo3/testing-framework keys the instance directory and the database on `substr(sha1(static::class), 0, 7)`. Each shard writes `--coverage-php=<dir>/<class>.cov`, and `phpunit/phpcov merge --clover coverage-functional.xml <dir>` merges them into the single file the shared workflow uploads. Run the merge with `php -d memory_limit=-1`.
- **`phpunit/phpcov` major follows PHPUnit**: the require-dev constraint `"phpunit/phpcov": "^10.0 || ^11.0 || ^12.0 || ^13.0"` resolves 10.0.1 with PHPUnit 11 (PHP 8.2 / TYPO3 13.4) and 13.2.0 with PHPUnit 13 (PHP 8.4 / TYPO3 14.3).
- **Count before merging**: refuse to merge when the number of `.cov` files differs from the number of test classes. A shard that wrote nothing silently shrinks the report.
- **PCOV, if used**: set `-d pcov.directory=<root>/Classes`, because PCOV falls back to the working directory (vendor included) when there is no `src/`, `lib/` or `app/`. In shards that collect no coverage, pass `-d pcov.enabled=0`: with PCOV loaded and enabled, non-coverage cells took 362–569 s instead of 197–381 s. `XDEBUG_MODE=off` has no PCOV counterpart.
- **Run the coverage path in the required cells** of pull requests and the merge queue, not only on schedule: some failures appear only with coverage enabled. Example: a `#[CoversClass]` naming a class excluded from `<source>` raises "is not a valid target for code coverage" once per test (see [ci-debugging.md](ci-debugging.md), "`CoversClass` on a coverage-excluded class"). With `failOnWarning` it failed every functional cell of a manual `main` run while all PR checks had been green, because the PR path ran without coverage. The split described under "A coverage drop can be an expired flag" below (PR runs without coverage) misses exactly these failures.
- **Reference implementation**: netresearch/t3x-nr-llm [`Build/Scripts/ciFunctionalParallel.sh`](https://github.com/netresearch/t3x-nr-llm/blob/main/Build/Scripts/ciFunctionalParallel.sh) and `.github/workflows/ci.yml`.
## E2E Testing in CI
> **IMPORTANT: Do NOT use DDEV in CI!**
>
> DDEV is for local development only. Use GitHub Services + PHP built-in server for E2E tests in CI.
### Why NOT DDEV in CI?
| Issue | Impact |
|-------|--------|
| **Slow startup** | 2-3+ minutes for Docker orchestration |
| **Complexity** | Docker-in-Docker, networking, volumes |
| **Resource heavy** | Multiple containers exceed runner limits |
| **Fragile** | Port conflicts, DNS issues, cert problems |
| **Non-standard** | TYPO3 Core uses direct PHP, not DDEV |
### Correct E2E CI Pattern: GitHub Services
```yaml
# .github/workflows/e2e.yml
name: E2E Tests
on: [push, pull_request]
jobs:
e2e:
runs-on: ubuntu-latest
timeout-minutes: 20
# GitHub Services - database container
services:
db:
image: mariadb:11.4
env:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: typo3
MYSQL_CHARSET: utf8mb4
MYSQL_COLLATION: utf8mb4_unicode_ci
ports:
- 3306:3306
options: >-
--health-cmd="healthcheck.sh --connect --innodb_initialized"
--health-interval=10s
--health-timeout=5s
--health-retries=5
steps:
- uses: actions/checkout@v4
- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.4'
extensions: mysqli, pdo_mysql, gd, intl
- name: Install Composer dependencies
run: composer install --prefer-dist --no-progress
- name: Setup TYPO3
run: |
mkdir -p .Build/Web/typo3conf
cat > .Build/Web/typo3conf/LocalConfiguration.php << 'EOF'
<?php
return [
'DB' => ['Connections' => ['Default' => [
'driver' => 'mysqli',
'host' => '127.0.0.1',
'dbname' => 'typo3',
'user' => 'root',
'password' => 'root',
]]],
'SYS' => [
'encryptionKey' => '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef',
'trustedHostsPattern' => 'localhost|127\\.0\\.0\\.1',
],
];
EOF
# Wait for database
for i in {1..30}; do
mysqladmin ping -h127.0.0.1 -uroot -proot --silent 2>/dev/null && break
sleep 2
done
.Build/bin/typo3 extension:setup --no-interaction
.Build/bin/typo3 backend:user:create --username=admin --password='Joh316!!' --admin --no-interaction
.Build/bin/typo3 cache:flush
- uses: actions/setup-node@v4
with:
node-version: '20' # Use LTS
- name: Install Playwright
run: |
npm ci
npx playwright install --with-deps chromium
# PHP built-in server (NOT DDEV)
- name: Start PHP server
run: |
php -S 0.0.0.0:8080 -t .Build/Web > /tmp/php-server.log 2>&1 &
for i in $(seq 1 30); do
curl -sf http://localhost:8080/typo3/ > /dev/null 2>&1 && break
sleep 1
done
- name: Run Playwright tests
env:
TYPO3_BASE_URL: http://localhost:8080
run: npm run test:e2e
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: Tests/E2E/Playwright/reports/
```
### Dual-Mode Playwright Configuration
Support both local (DDEV) and CI (localhost) environments:
```typescript
// playwright.config.ts
export default defineConfig({
use: {
// DDEV for local, CI sets TYPO3_BASE_URL=http://localhost:8080
baseURL: process.env.TYPO3_BASE_URL || 'https://my-extension.ddev.site',
ignoreHTTPSErrors: true, // For DDEV self-signed certs
},
});
```
### runTests.sh Integration
The `runTests.sh` script should support both modes:
```bash
run_playwright_tests() {
local typo3_base_url="${TYPO3_BASE_URL:-https://my-extension.ddev.site}"
# Check if TYPO3 is accessible (use -k for https/DDEV)
local curl_opts="-s"
[[ "${typo3_base_url}" == https://* ]] && curl_opts="-sk"
if ! curl ${curl_opts} "${typo3_base_url}/typo3/" > /dev/null 2>&1; then
if [[ "${PLAYWRIGHT_FORCE:-0}" != "1" ]]; then
echo "Error: TYPO3 not responding at ${typo3_base_url}"
echo "Set PLAYWRIGHT_FORCE=1 to override."
exit 1
fi
fi
export TYPO3_BASE_URL="${typo3_base_url}"
npm run test:e2e
}
```
## GitLab CI
### Basic Pipeline
Create `.gitlab-ci.yml`:
```yaml
variables:
COMPOSER_CACHE_DIR: ".composer-cache"
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "typo3_test"
cache:
key: "$CI_COMMIT_REF_SLUG"
paths:
- .composer-cache/
stages:
- lint
- analyze
- test
.php:
image: php:${PHP_VERSION}-cli
before_script:
- apt-get update && apt-get install -y git zip unzip
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install --no-progress
lint:
extends: .php
stage: lint
variables:
PHP_VERSION: "8.2"
script:
- composer ci:test:php:lint
phpstan:
extends: .php
stage: analyze
variables:
PHP_VERSION: "8.2"
script:
- composer ci:test:php:phpstan
cgl:
extends: .php
stage: analyze
variables:
PHP_VERSION: "8.2"
script:
- composer ci:test:php:cgl
unit:8.1:
extends: .php
stage: test
variables:
PHP_VERSION: "8.1"
script:
- composer ci:test:php:unit
unit:8.2:
extends: .php
stage: test
variables:
PHP_VERSION: "8.2"
script:
- composer ci:test:php:unit
coverage: '/^\s*Lines:\s*\d+.\d+\%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
functional:8.2:
extends: .php
stage: test
variables:
PHP_VERSION: "8.2"
typo3DatabaseDriver: "mysqli"
typo3DatabaseHost: "mysql"
typo3DatabaseName: "typo3_test"
typo3DatabaseUsername: "root"
typo3DatabasePassword: "root"
services:
- mysql:8.0
script:
- composer ci:test:php:functional
```
### Multi-Database Testing
```yaml
.functional:
extends: .php
stage: test
variables:
PHP_VERSION: "8.2"
script:
- composer ci:test:php:functional
functional:mysql:
extends: .functional
variables:
typo3DatabaseDriver: "mysqli"
typo3DatabaseHost: "mysql"
typo3DatabaseName: "typo3_test"
typo3DatabaseUsername: "root"
typo3DatabasePassword: "root"
services:
- mysql:8.0
functional:postgres:
extends: .functional
variables:
typo3DatabaseDriver: "pdo_pgsql"
typo3DatabaseHost: "postgres"
typo3DatabaseName: "typo3_test"
typo3DatabaseUsername: "postgres"
typo3DatabasePassword: "postgres"
services:
- postgres:15
before_script:
- apt-get update && apt-get install -y libpq-dev
- docker-php-ext-install pdo_pgsql
functional:sqlite:
extends: .functional
variables:
typo3DatabaseDriver: "pdo_sqlite"
```
## Best Practices
### 1. Fast Feedback Loop
Order jobs by execution time (fastest first):
```yaml
stages:
- lint # ~30 seconds
- analyze # ~1-2 minutes (PHPStan, CGL)
- unit # ~2-5 minutes
- functional # ~5-15 minutes
- acceptance # ~15-30 minutes
```
### 2. Fail Fast
```yaml
strategy:
fail-fast: true # Stop on first failure
matrix:
php: ['8.1', '8.2', '8.3', '8.4']
```
### 3. Parallel Execution
```yaml
# GitHub Actions - parallel jobs
jobs:
lint: ...
phpstan: ...
unit: ...
# All run in parallel
# GitLab CI - parallel jobs
test:
parallel:
matrix:
- PHP_VERSION: ['8.1', '8.2', '8.3']
```
### 4. Cache Dependencies
GitHub Actions:
```yaml
- uses: actions/cache@v3
with:
path: ~/.composer/cache
key: ${{ runner.os }}-composer-${{ hashFiles('**/composer.lock') }}
```
GitLab CI:
```yaml
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .composer-cache/
```
### 5. Matrix Testing
Test critical combinations:
```yaml
strategy:
matrix:
include:
# Minimum supported versions
- php: '8.1'
typo3: '12.4'
# Current stable
- php: '8.2'
typo3: '12.4'
# Latest versions
- php: '8.3'
typo3: '13.0'
```
### 6. Artifacts and Reports
```yaml
- name: Archive test results
if: failure()
uses: actions/upload-artifact@v3
with:
name: test-results
path: |
var/log/
typo3temp/var/tests/
```
### 7. Notifications
GitHub Actions:
```yaml
- name: Slack Notification
if: failure()
uses: rtCamp/action-slack-notify@v2
env:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
```
## Quality Gates
### Required Checks
Define which checks must pass:
GitHub:
```yaml
# .github/branch-protection.json
{
"required_status_checks": {
"strict": true,
"contexts": [
"lint",
"phpstan",
"unit (8.2)",
"functional (8.2, mysqli)"
]
}
}
```
GitLab:
```yaml
# .gitlab-ci.yml
unit:8.2:
only:
- merge_requests
allow_failure: false # Required check
```
### Coverage Driver Issues
When PHPUnit config files include `<coverage>` sections, tests will fail if no coverage driver (xdebug/pcov) is available. Add `--no-coverage` flag when coverage is disabled:
```yaml
- name: Run unit tests
run: |
if [ "$COVERAGE_ENABLED" = "true" ]; then
vendor/bin/phpunit -c Build/phpunit/UnitTests.xml
else
vendor/bin/phpunit -c Build/phpunit/UnitTests.xml --no-coverage
fi
```
**Common error**: `PHPUnit\Framework\InvalidArgumentException: No code coverage driver available`
**Solution**: Either install a coverage driver (xdebug, pcov) or pass `--no-coverage`:
```bash
# Install pcov for faster coverage
pecl install pcov
# Or disable coverage in CI
vendor/bin/phpunit --no-coverage
```
### Coverage Requirements
```yaml
- name: Check code coverage
run: |
coverage=$(vendor/bin/phpunit --coverage-text | grep "Lines:" | awk '{print $2}' | sed 's/%//')
if (( $(echo "$coverage < 80" | bc -l) )); then
echo "Coverage $coverage% is below 80%"
exit 1
fi
```
### Debugging "green before, red now" on unchanged code
TYPO3 extensions are libraries and **do not commit `composer.lock`** — so every
CI run does a fresh `composer update` and resolves dependencies (including the
dev toolchain: PHPStan, Rector, php-cs-fixer, testing-framework) to the latest
versions allowed by `composer.json`. A green pipeline can therefore turn red with
**no change to your code**, simply because an upstream package published a new
release between runs.
When a check fails on a commit (or PR) that previously passed with identical
code, suspect a fresh upstream release before touching your own code:
```bash
# Which version did the failing run install vs. a previous green run?
# composer logs full package names, e.g. "Installing phpstan/phpstan (2.2.2)".
gh run view <run-id> --log | grep -iE "Installing (phpstan|rector|friendsofphp|typo3/testing-framework)/"
# Cross-check release dates on Packagist. The /p2/ endpoint returns versions
# newest-first, so the first entries are the most recent releases.
curl -s https://repo.packagist.org/p2/phpstan/phpstan.json \
| php -r '$d = json_decode(file_get_contents("php://stdin"), true); foreach (array_slice($d["packages"]["phpstan/phpstan"], 0, 6) as $v) { echo $v["version"] . " " . $v["time"] . PHP_EOL; }'
```
Reproduce deterministically by pinning the suspect version locally
(`composer require --dev "phpstan/phpstan:X.Y.Z"`), confirm it fails, then revert
to the floating constraint after the fix. Note that a newer analyzer release is
often a **true positive** surfacing a latent bug — fix the code, don't pin to
escape it. Pin only as a temporary, documented escape hatch when the release is
genuinely broken — and exclude just the broken release *in addition to* your
normal range (e.g. `^1.12,!=1.12.3`), never a bare `!=1.12.3` (which would also
permit unexpected major upgrades), so future fixes still flow in.
## Environment-Specific Configuration
### Development Branch
```yaml
on:
push:
branches: [ develop ]
# Run all checks, allow failures
jobs:
experimental:
continue-on-error: true
strategy:
matrix:
php: ['8.4'] # Experimental PHP version
```
### Production Branch
```yaml
on:
push:
branches: [ main ]
# Strict checks only
jobs:
tests:
strategy:
fail-fast: true
matrix:
php: ['8.2'] # LTS version only
```
### Pull Requests
```yaml
on:
pull_request:
# Full test matrix
jobs:
tests:
strategy:
matrix:
php: ['8.1', '8.2', '8.3', '8.4']
database: ['mysqli', 'postgres']
```
## Netresearch CI Integration
Netresearch TYPO3 extensions use reusable workflows from `netresearch/typo3-ci-workflows` instead of defining CI steps directly in project repositories.
### Core Principle
**NEVER add direct GitHub Actions steps (checkout, setup-php, composer install, etc.) to project workflows.** All CI logic is centralized in reusable workflows provided by `netresearch/typo3-ci-workflows`. Projects only configure inputs and call the shared workflows.
### Single Dev Dependency
```bash
composer require --dev netresearch/typo3-ci-workflows:^1.1
```
This single package transitively provides all quality tools (PHPStan, php-cs-fixer, Rector, PHPUnit, Infection, phpat, captainhook, etc.), shared configurations, and reusable GitHub Actions workflows.
### Workflow Configuration (.github/workflows/ci.yml)
```yaml
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
ci:
uses: netresearch/typo3-ci-workflows/.github/workflows/ci.yml@main
with:
php-versions: '["8.2", "8.3", "8.4", "8.5"]'
typo3-versions: '["^13.4"]'
```
### Push Trigger: Restrict to main
The `push` trigger MUST be restricted to `branches: [main]` to avoid duplicate CI runs. Without this restriction, pushing to a branch with an open PR triggers both a `push` event and a `pull_request` event, resulting in duplicate workflow runs:
```yaml
# Correct: push only on main, PR on all branches
on:
push:
branches: [main]
pull_request:
# Wrong: push on all branches causes duplicate runs with PRs
on:
push:
pull_request:
```
### Test Matrix
The reusable workflow handles the full test matrix based on inputs:
| Dimension | Values | Notes |
|-----------|--------|-------|
| PHP | 8.2, 8.3, 8.4, 8.5 | All versions TYPO3 v13 supports |
| TYPO3 | ^13.4 | Current LTS |
| Database | pdo_sqlite (default) | Configurable per project |
### PHPStan Extensions Auto-Discovery
When using `netresearch/typo3-ci-workflows`, PHPStan extensions (phpstan-strict-rules, phpstan-typo3, phpstan-phpunit, etc.) are auto-discovered by `phpstan/extension-installer`. Do NOT manually include extension neon files in your `phpstan.neon`:
```neon
# Correct: only include shared config and baseline
includes:
- %currentWorkingDirectory%/.Build/vendor/netresearch/typo3-ci-workflows/config/phpstan/phpstan.neon
- phpstan-baseline.neon
parameters:
paths:
- ../Classes
- ../Tests/Architecture
```
```neon
# Wrong: manual includes cause duplicate registration errors
includes:
- %currentWorkingDirectory%/.Build/vendor/phpstan/phpstan-strict-rules/rules.neon
- %currentWorkingDirectory%/.Build/vendor/saschaegerer/phpstan-typo3/extension.neon
```
**Exception:** When using git worktrees with `composer install --no-plugins`, use the explicit includes file `includes-no-extension-installer.neon` instead (see quality-tools.md for details).
### What the Reusable Workflow Runs
The CI workflow executes these checks (projects do not need to define them):
1. **Linting** -- PHP syntax validation
2. **Code style** -- php-cs-fixer dry-run
3. **Static analysis** -- PHPStan at level 10
4. **Unit tests** -- across PHP version matrix
5. **Functional tests** -- across PHP version matrix
6. **Mutation testing** -- Infection PHP
7. **Architecture tests** -- phpat rules
8. **Security audit** -- `composer audit`
## Coverage Reporting Pitfalls
### A "coverage drop" can be an expired flag, not missing tests
Codecov merges per-flag sessions (`unit`, `functional`, …) into the project
number. Flags with `carryforward: true` reuse the last uploaded session — and
when that session **expires** (no fresh upload within the retention window),
the flag silently contributes 0/0 lines and the project number collapses to
the remaining flags. The repo then *looks* like it lost coverage with no code
change.
Before writing tests to close a coverage gap, verify each flag is alive:
```bash
# 0/0 lines => the flag expired; the gap is a reporting artifact
curl -s "https://api.codecov.io/api/v2/github/<org>/repos/<repo>/report/?flag=functional" \
| jq '.totals | {coverage, hits, lines}'
```
Restore an expired flag by re-running the upload path — in the standard
Netresearch split (PR runs = parallel, no coverage; `schedule`/
`workflow_dispatch` = serial + Xdebug + upload) that is one dispatch:
```bash
gh workflow run ci.yml --ref main # ~15 min; matrix cells upload the flag
```
A PR path without coverage cannot see failures that exist only under
coverage; see "Sharded functional tests with coverage" above for running the
coverage path in the required cells.
Real case: a "coverage >= 80%" goal opened against a repo reading 68.6% —
the functional flag had expired; the true full-suite number was 86%. The
dispatch met the goal; the phantom gap would have cost days of test-writing.
Corollary: measure locally only for per-file gap analysis — a full functional
suite under Xdebug on WSL2 projects to hours; the CI dispatch is the fast path.
### On a PR, the same drop is usually an upload still in flight
The expired-flag case above is the *repo-level* variant. On a pull request the
identical symptom — `codecov/project` red, coverage down several points — far
more often means one flag's upload for **this head commit** has not landed yet
at the moment Codecov computed the status. Nothing is wrong and nothing needs
doing; the status recomputes when the upload arrives.
Three signals identify it, and all three must hold:
- **`codecov/patch` is green** and says *"Coverage not affected"*. A real
regression from the PR's own diff would show up here first.
- **The head number equals one flag's standalone coverage.** The project total
has collapsed to the flags that *did* upload. Compare against
`api.codecov.io/.../report/?flag=<flag>` per flag — if head% matches `unit`
to two decimals, only `unit` is in the report.
- **The missing flag's own report is healthy** — non-zero `lines`, not `0/0`.
`0/0` means expired (previous section); healthy-but-absent means in flight.
```bash
# head total vs each flag standalone — the match names the flag that uploaded
for f in unit functional acceptance; do
printf '%s: ' "$f"
curl -s "https://api.codecov.io/api/v2/github/<org>/repos/<repo>/report/?flag=$f" \
| jq -c '.totals | {coverage, lines}'
done
```
Observed: `codecov/project` 89.60% (−4.65%) with `codecov/patch` reporting
"Coverage not affected" on a release PR that changed only a version literal and
Markdown. 89.60% was `unit` (89.54%) alone; `functional` was healthy at 39.05%
over the same 1444 lines but its matrix cells had not finished. The status went
green on its own once they did.
The window exists only when the repo's `codecov.yml` has drifted from the
shipped `assets/codecov.yml`, which sets `carryforward: true` on **every**
uploading flag. A flag without it contributes nothing until its own upload for
that commit lands; a flag with it falls back to the previous session and the
project number stays stable. So check the repo's `flags:` block against the
asset — the incident above was a repo carrying `carryforward` on `unit` only.
Until the config is aligned, treat a project-only drop on a PR as pending, not
as a finding — do not re-run CI, adjust the target, or start writing tests for
it.
### Stale coverage cache fails the suite with zero failing tests
PHPUnit's static-analysis cache (`.Build/cache/phpunit/code-coverage/`) is
written by the container user of whoever ran coverage last. A later run under
a different uid gets `file_put_contents(): Permission denied` **warnings** —
and with `failOnWarning="true"` the suite exits 1 with 0 failures/errors.
Remove the cache dir before coverage runs when containers/users alternate:
```bash
rm -rf .Build/cache/phpunit/code-coverage
```
## Resources
- [GitHub Actions Documentation](https://docs.github.com/actions)
- [GitLab CI Documentation](https://docs.gitlab.com/ee/ci/)
- [TYPO3 Tea Extension CI](https://github.com/TYPO3BestPractices/tea/tree/main/.github/workflows)
- [shivammathur/setup-php](https://github.com/shivammathur/setup-php)
- [netresearch/typo3-ci-workflows](https://github.com/netresearch/typo3-ci-workflows)
## No-lock libraries resolve per PHP matrix leg — verify on each
A library without `composer.lock` resolves dependencies fresh per CI run AND per PHP version — each matrix leg can install a different dependency set, and the local default PHP's resolution is not representative. A PHPStan baseline generated on local PHP 8.5 (which resolved Symfony 8) would not have matched the 8.1/8.2 CI legs (Symfony 6.4/7.4). Before claiming a no-lock library green, re-resolve per CI PHP version: `composer config platform.php 8.1.99 && composer update --with-all-dependencies`, run the checks, repeat per leg.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |