netresearch/typo3-testing-skill / skills/typo3-testing
typo3-testing
FlaggedUse when reproducing defects as failing tests, testing template or rendered output changes, confirming changes on a running TYPO3 instance, or setting up TYPO3 extension tests: unit/functional/E2E, PHPUnit 11-13, mutation testing, final-class mocks (v14), CI matrix across TYPO3 12/13/14.3 LTS, typo3-ci-workflows meta-package, CI failures. Also: transaction rollback, worker crashes, competing requests, testing-framework setup, test matrix, integration/e2e testing, coverage, test generation.
references/ci-debugging.md
View on GitHub<!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
<!-- SPDX-FileCopyrightText: Netresearch DTT GmbH -->
# Debugging CI Test Failures
## Multi-Version Error Analysis
When tests fail in CI across multiple TYPO3 versions, **always check error messages from ALL matrix combinations** (v13 AND v14, all PHP versions). Different TYPO3 versions often fail with completely different errors for the same root cause.
### Common Error Pairs
| v13 Error | v14 Error | Root Cause |
|-----------|-----------|------------|
| `parseFunc without any configuration` | `No valid attribute "applicationType"` | Missing TSFE bootstrap |
| Method signature mismatch | Missing interface method | API change between versions |
| Deprecated function warning | Fatal: undefined method | Removed API |
| Test passes | `RuntimeException` in DI container | Singleton resolution order changed |
## Debugging Checklist
1. **Get error counts per matrix:**
```bash
gh run view <RUN_ID> --log-failed 2>&1 | grep "There were"
```
2. **Compare v13 vs v14 errors** — different errors often mean different root causes
3. **Check regression scope:**
- Only your new tests fail → your test setup is incomplete
- Existing tests also fail → your change has side effects (e.g., `$GLOBALS` pollution)
4. **Get detailed errors per version:**
```bash
gh run view <RUN_ID> --log-failed 2>&1 | grep "^build.*13.4.*8.5.*Functional.*) " | head -20
gh run view <RUN_ID> --log-failed 2>&1 | grep "^build.*14.0.*8.5.*Functional.*) " | head -20
```
## Common Pitfalls
### `$GLOBALS['TYPO3_REQUEST']` Pollution
Setting `$GLOBALS['TYPO3_REQUEST']` in `setUp()` affects ALL tests in the class:
- **v14:** Requires `applicationType` attribute — missing it causes `RuntimeException` in PageRenderer/DI container resolution (63+ errors)
- **v13:** Enables additional processing paths — existing test assertions may no longer match (7+ failures)
- **Both:** the attribute value is the int bitmask `SystemEnvironmentBuilder::REQUESTTYPE_FE`/`_BE`, not the `ApplicationType` enum case — `ApplicationType::fromRequest()` guards with `is_int()` and throws `RuntimeException` 1606222812 otherwise
**Fix:** Set the global only in specific test methods that need it, with `try/finally` cleanup:
```php
$GLOBALS['TYPO3_REQUEST'] = $this->request
->withAttribute('applicationType', SystemEnvironmentBuilder::REQUESTTYPE_FE);
try {
// test code
} finally {
unset($GLOBALS['TYPO3_REQUEST']);
}
```
### Functional Tests Cannot Call `parseFunc()` with TypoScript References
`ContentObjectRenderer::parseFunc($html, null, '< lib.parseFunc_RTE')` requires:
- TypoScript configuration loaded (v13: `LogicException`)
- Full request with `applicationType` (v14: `RuntimeException`)
- `$GLOBALS['TYPO3_REQUEST']` for child cObj instances
**Solution:** Use unit tests (mock `parseFunc`) + E2E tests (real frontend). See `functional-testing.md` for details.
### Test Isolation Between Matrix Entries
Each matrix entry (PHP version × TYPO3 version) runs independently. A test passing on `8.2 + v13` but failing on `8.5 + v14` indicates version-specific behavior, not flakiness.
## Run each gate in the image its own CI job uses
A pipeline rarely runs every job in one image, and the differences are not
cosmetic. In one repository `test:php`, `test:rector` and `test:phpstan` run in
`ghcr.io/devgine/composer-php:v2-php8.4-alpine` while `test:unit` runs in plain
`php:8.4` and installs what it needs first:
```yaml
before_script:
- apt-get install git unzip zlib1g-dev libzip-dev -yqq
- docker-php-ext-install zip
```
The alpine image has **no `ext-zip`**. Reproducing the suite there makes every
zip-touching test error out locally while CI is green, and the failure names the
test, not the image — so it reads as a broken test. `composer install
--ignore-platform-reqs`, which these pipelines use, removes the one signal that
would have said otherwise.
Read `.gitlab-ci.yml` / the workflow file for the `image:` **and** the
`before_script:` of the specific job before reproducing it, and mirror both. A
throwaway Dockerfile that copies the job's `before_script` is worth it as soon as
you run the suite more than twice:
```dockerfile
FROM php:8.4
RUN apt-get update -yqq \
&& apt-get install -yqq git unzip zip zlib1g-dev libzip-dev \
&& docker-php-ext-install zip
```
Also mirror the flags: the phpstan job runs `php -d memory_limit=2G`, and without
it PHPStan dies with *reached configured PHP memory limit: 128M* and reports
"Found 2 errors" that have nothing to do with the code.
### Without Docker: match the job's PHP, not just its packages
When Docker is not available, the host PHP is usually newer than the job's, and
an old toolchain does not survive that. With a PHP 8.1 job (PHPStan 1.10,
Rector 0.17) run under PHP 8.5, Rector died with
`Type of PHPStan\BetterReflection\…\ReflectionClass::IS_READONLY must be
compatible`, and PHPStan reported 2 errors where the job reported 12. A local
run with the wrong runtime measures the runtime.
A static PHP CLI of the job's minor version needs no installation:
```bash
d=/absolute/scratch/dir # outside the extension checkout
curl -fsL -o "$d/php81.tar.gz" \
https://dl.static-php.dev/static-php-cli/common/php-8.1.34-cli-linux-x86_64.tar.gz
tar xzf "$d/php81.tar.gz" -C "$d" && "$d/php" -v && "$d/php" -m
```
Resolve the packages for that version too. In the extension directory, write a
copy of `composer.json` that pins the platform and point Composer at it.
Composer then writes `composer.local.lock`; keep both files out of git
(`.git/info/exclude`) and delete the lock before each comparison, or the
install reuses it and prints no `Locking` lines:
```bash
jq '.config.platform.php="8.1.30"' composer.json > composer.local.json
rm -f composer.local.lock
COMPOSER=composer.local.json composer install --ignore-platform-req='ext-*'
```
Then compare the `Locking …` lines of that install with the job log's: an equal
set means the local run checks the same code with the same tools. Run each
tool through the static binary (`"$d/php" vendor/bin/phpstan …`). A
`composer install` may also install GrumPHP git hooks, which call `php` from
`PATH`; prepend `"$d"` to `PATH` for commits, or the hook runs the new PHP.
Composer writes that package list to **stderr**, not stdout, so a capture with
`2>/dev/null` comes back empty and reads as "nothing resolved". Capture with
`2>&1`. Count `- Locking` lines only: on a resolve without a lock file and
without installed packages, every package appears once as `- Locking` and once
as `- Installing`, so counting all package lines (every line starting with two
spaces and a dash) gives exactly twice the number. With an existing lock file
or vendor directory the two lists differ; for a complete count read the
resolved `composer.lock` instead. Measured with
Composer 2.11.2, `composer update --dry-run` on a project with 27 packages: 0
lines with `2>/dev/null`; with `2>&1`, 54 package lines and 27 `- Locking`
lines. Twelve repositories that all report an empty list are one broken
pipeline, not twelve broken repositories.
## Testing-Framework Version Mapping
| testing-framework | PHPUnit | TYPO3 Versions |
|-------------------|---------|----------------|
| v8 | 10 | 12.4, 13.4 |
| v9 | 11 | 13.4, 14.0+ |
## PHPUnit 11 Compatibility Issues
### Final TestCase Constructor
PHPUnit 11 makes `TestCase::__construct()` final. Extensions that override the constructor will fail:
```
Cannot override final method PHPUnit\Framework\TestCase::__construct()
```
**Fix:** Replace constructor-based initialization with property declarations:
```php
// ❌ PHPUnit 11: Fatal error
abstract class ExtensionTestCase extends FunctionalTestCase
{
public function __construct(string $name = '')
{
parent::__construct($name);
$this->coreExtensionsToLoad = ['install'];
$this->testExtensionsToLoad = ['vendor/extension'];
}
}
// ✅ Works with both PHPUnit 10 and 11
abstract class ExtensionTestCase extends FunctionalTestCase
{
protected array $coreExtensionsToLoad = ['install'];
protected array $testExtensionsToLoad = ['vendor/extension'];
}
```
### Share these defaults through a base class, not a trait
A trait that declares `$coreExtensionsToLoad` or `$testExtensionsToLoad` with a value different from `FunctionalTestCase`'s default is a fatal error at class load on PHP 8.2, 8.3 and 8.4, and runs on PHP 8.5. Measured with the same property type and visibility in parent and trait and only the default differing (PHP 8.2.36, 8.3.37, 8.4.28: exit 255; PHP 8.5.13: exit 0):
```text
Fatal error: P and T define the same property ($x) in the composition of C. However, the definition differs and is considered incompatible. Class was composed in /w/t.php on line 4
```
So the trait passes on a local PHP 8.5 and fails every older cell of the matrix before a test runs. Overriding an inherited property in an abstract base class, as above, is legal on every version; a trait that carries only methods is fine. When you move per-class copies into one shared helper, choose the mechanism against the oldest PHP the extension supports and run the check on that version.
## PHPStan: "Call to an undefined static method …::assert…()"
`self::assertSame()` is correct, and TYPO3 CGL (php-cs-fixer) enforces it: PHPUnit declares its assertions `final public static` in `PHPUnit\Framework\Assert` (checked at PHPUnit 10.5.0, 11.5.0, 12.0.0 and 13.4.6).
When PHPStan reports `Call to an undefined static method <TestClass>::assertSame()` (`staticMethod.notFound`), look first for `Class <TestClass> extends unknown class PHPUnit\Framework\TestCase` (`class.notFound`) in the same run. Without that line, the assertion does not exist in the installed PHPUnit (PHPUnit 13 removed `assertContainsOnly()`, for example): change the assertion. With it, PHPStan could not load PHPUnit's `TestCase`. Two setups produce that:
| Cause | Fix |
|-------|-----|
| PHPStan 1.x analyses tests against PHPUnit 13 (`typo3/testing-framework` allows `^13` from 9.4.0) | `composer require --dev "phpstan/phpstan:^2"` |
| PHPUnit is not installed in the Composer project PHPStan runs from | Require it as a dev dependency of that project (`typo3/testing-framework` or `phpunit/phpunit`) |
Measured with PHPStan 1.12.34 (the last 1.x line): against PHPUnit 13.4.6 it reports both errors, with and without `phpVersion: 80100`; against PHPUnit 11.5.57 and 12.5.43 it reports neither. PHPStan 2.3.5 reports neither against PHPUnit 11.5.57 or 13.4.6, also with `typo3/testing-framework` 9.7.0, and reports both when PHPUnit is absent from its project.
Do not suppress the message with an `ignoreErrors` pattern on `Call to an undefined static method`. The pattern does not match the `extends unknown class` error, so the run stays red, and it hides that PHPStan analysed the test classes without PHPUnit.
## PHPStan exits 255: `Cannot set the alias loader, as it is already registered!`
`vendor/phpstan/phpstan/bin/phpstan` first loads the phar's own autoloader, then the Composer autoloader of its **working directory** (it reads that directory's `composer.json` and its `config.vendor-dir`), and only then the autoloader of the vendor tree the phar sits in (read in `bin/phpstan` inside `phpstan.phar` of PHPStan 2.3.7). When the working directory and the PHPStan binary belong to two different vendor trees, both project autoloaders are loaded.
That happens with a local runner that keeps a separate vendor tree per PHP/TYPO3 combination next to the extension's own `.Build/vendor` and starts PHPStan from the extension root. In one extension that layout made `typo3/class-alias-loader` register twice, and the process died before reading any configuration:
```text
Uncaught RuntimeException: Cannot set the alias loader, as it is already registered!
```
The exit code is 255. Two checks separate this from a broken `phpstan.neon`: `phpstan --version` fails the same way, and the error disappears when the root `.Build/vendor` is moved away. The runner breaks only for developers who also ran `composer install` in the root, and CI never sees it because CI does not use the local runner.
**Fix:** start PHPStan with the directory of the vendor tree it belongs to as the working directory, and give it a configuration with absolute paths so the changed working directory does not move the analysed scope. A configuration included from `netresearch/typo3-ci-workflows` resolves its `bootstrapFiles` through `%currentWorkingDirectory%/.Build/vendor` and its `ignoreErrors` paths through `%currentWorkingDirectory%/Tests/…` and `%currentWorkingDirectory%/Classes/…`, so rewrite every `%currentWorkingDirectory%` path in a copy of that included file too, not only in the outer neon; otherwise the relocated run reports errors the root run suppresses. Verify both states: with and without a root `.Build/vendor`.
## A resolve outside the test runner is no PHPStan baseline
A `composer update` run in a plain `composer` container instead of through `runTests.sh` leaves three artefacts that look like code findings:
- `.Build` is owned by root, and the runner, which runs as your user, then fails with `Directory ".Build/public/typo3conf/ext" could not be created`. See "Linux --user Flag" in [test-runners.md](test-runners.md).
- In a git worktree, CaptainHook's installer exits 1 with `invalid .git path`. The packages are installed; the exit code is the hook installer's. See "captainhook + git worktree" in [quality-tools.md](quality-tools.md).
- `phpstan/extension-installer` may not register every extension. One run left `ergebnis/phpstan-rules` unregistered, and PHPStan stopped with `Unexpected item 'parameters › ergebnis'`; a second `composer install` did not change it.
The obvious cross-check, PHPStan on `main`, is worth as little when `main`'s `.Build` is stale. In one extension it still held a dependency one minor below the constraint in `composer.json`, so the errors it reported belonged to a version the package no longer supports.
When neither number comes from a resolve you trust, report no PHPStan number at all. Run the behavioural suites (unit, functional) locally, say that the static analysis was left to CI, and let CI resolve from the declared constraints on a fresh checkout.
## Archived TYPO3-CI GitHub Actions
Several TYPO3 CI GitHub Actions have been archived and their Docker images return 403 Forbidden:
| Action | Status | Replacement |
|--------|--------|-------------|
| `TYPO3-CI-Xliff-Lint` | Archived (2021) | DIY `xmllint --schema xliff-core-1.2-strict.xsd` or remove if no `.xlf` files |
| Other `TYPO3-Continuous-Integration/*` | Check individually | May need replacement |
**Before adding an XLIFF linter:** Verify the extension actually has `.xlf` files:
```bash
find . -name '*.xlf' -not -path './.Build/*'
```
Many extensions don't ship translations and the CI job was added as boilerplate.
## Test Fixture Isolation from TYPO3 Core
When tests depend on TYPO3 core class docblocks (e.g., testing documentation generation), **use local fixture classes** instead:
**Problem:** TYPO3 core changes docblock wording between versions (e.g., "that" → "which"), causing test assertion failures across the matrix.
**Solution:** Create controlled fixture classes in `Tests/Functional/Fixtures/Extensions/`:
```php
// Local fixture with stable, controlled docblock
namespace TYPO3Tests\ExampleExtension;
class PropertyExample
{
/**
* This is set to the language that is currently running
*/
public string $lang = 'default';
}
```
Update test config to reference the fixture class instead of the core class. This decouples tests from core docblock changes across TYPO3 versions.
## phpDocumentor Version Differences in Tests
phpDocumentor v8 and v9 differ in generic type rendering:
- **v8:** Preserves original spacing: `array<string,string>`
- **v9:** Normalizes with spaces: `array<string, string>`
**Fix:** Normalize generic type spacing in code that processes phpDoc output:
```php
// Strip spaces after commas inside angle brackets
preg_replace_callback('/<[^>]+>/', static function (array $match): string {
return str_replace(', ', ',', $match[0]);
}, $type);
```
**Related bug:** Never use `explode(' ', $returnComment, 2)` to split type from description when generic types are involved — types like `array<string, string>` contain internal spaces. Use bracket-depth-aware parsing instead.
## `CoversClass` on a coverage-excluded class: red in CI, green locally
`#[CoversClass(X::class)]` where `X` is excluded from the coverage source set in `Build/phpunit.xml` emits *"Class X is not a valid target for code coverage"* — one PHPUnit warning per test, and with `failOnWarning="true"` that is a red build. The trap: the warning only exists when coverage is **enabled**. `runTests.sh -s unit` runs without coverage and reports zero warnings; CI runs with Xdebug and sees them all. Reproduce locally with `runTests.sh -s unitCoverage` before pushing, whenever you add `CoversClass` attributes or touch the coverage include/exclude lists.
## codecov/patch fails on lines with `assert()`
`shivammathur/setup-php` applies the production `php.ini` by default, and that sets `zend.assertions = -1` (`php.ini-production`; the development file sets `1`). With `-1`, every `assert()` is compiled away, which has two effects: the assertions check nothing in CI, and the coverage report can still list those lines as executable statements that never get a hit. Then every pull request whose diff touches an `assert()` line fails the codecov/patch target, whatever its tests do.
The signature, as observed in one extension: the `assert()` line appears in the Clover report as `<line num="…" type="stmt" count="0"/>` although the method around it is demonstrably tested.
**Fix:** set `ini-values: zend.assertions=1` on the `setup-php` step of every PHPUnit job. The reusable `netresearch/typo3-ci-workflows` `ci.yml` does this for its test jobs. Reproduce locally with the same setting, e.g. `php -d zend.assertions=1 .Build/bin/phpunit …`, or with that `-d` flag added to the PHP options of your `runTests.sh`.
Before reading any diff-coverage result, check that at least one file in the Clover report has hits. A report in which every file has zero hits means the run broke (for example, every functional test errored on a missing `mysqli`) and the report was written anyway; it does not mean nothing is covered.
## A PHP extension segfaults in functional tests
- **Recognise it**: exit code 139 and "Segmentation fault (core dumped)". That is not PHP memory exhaustion ("Allowed memory size exhausted"), and not exit 137 ("Killed"), which is a SIGKILL: often the out-of-memory killer, but confirm it in the container or kernel log (`dmesg`, `docker inspect --format '{{.State.OOMKilled}}'`) before calling it one. Rule out the opcache JIT first, which crashes the same way: see "Functional suites: run WITHOUT JIT" in [test-runners.md](test-runners.md).
- **Find the core dump**: the testing framework changes into the TYPO3 test instance directory, so core dumps of functional tests land in `.Build/Web/typo3temp/var/tests/functional-<hash>/`, not in the working directory. Enable them with `docker run --ulimit core=-1 …` (core_pattern `core`).
- **Read the core dump instead of running under gdb**: the crash may disappear under gdb (it did here). `CORE=.Build/Web/typo3temp/var/tests/functional-<hash>/core.<pid>; gdb -batch -ex 'bt 12' -ex 'info locals' "$(command -v php)" "$CORE"`
- **Get function names**: build the extension from the release tarball with debug info. For Xdebug:
```bash
curl -sSLO https://xdebug.org/files/xdebug-<v>.tgz
tar xzf xdebug-<v>.tgz && cd xdebug-<v>
phpize && CFLAGS="-g -O2" ./configure --enable-xdebug && make
cp modules/xdebug.so "$(php-config --extension-dir)/"
```
In the TYPO3 core-testing image install the build dependencies first: `apk add $PHPIZE_DEPS linux-headers`.
- **Bisect versions**: build one image per version (`pecl install -f xdebug-<v>` as root, then `docker commit`), run the failing test N times per image as the normal user, and compare the exit codes.
- **Narrow to one method**: `--list-tests`, then `--filter '/::<method>( |$)/'`.
Example: Xdebug 3.5.x with `--path-coverage`, see "Branch and path coverage" in [ci-cd.md](ci-cd.md).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |