Merge conflict Gerrit cannot merge this change. It needs a rebase onto main.

Review main Work in progress Stale

95337[WIP][TASK] Evaluate phpunit parallel test execution

Stefan Bürk CR 0 V 0 Hard +314 −173 updated 08/15/26, 12:49

Commit message

Sebastian Bergmann asked the core team to try out his draft pull request
"Parallel test execution" and to report back whether it is useful for
TYPO3 core development. We are glad to pick that up and happy to give
feedback.

This is a test commit to gather that feedback, it is not meant to be
merged. It answers the question with a clear yes: with two fixes in the
pull request and a handful of test cases adjusted on our side, the whole
unit test suite passes in parallel in about 8 seconds instead of 25, and
the functional chunks pass as well.

composer.json resolves phpunit/phpunit from a fork of the pull request
branch, which carries two commits fixing the defects described below.
The repository entry sets "no-api", because the github api answers
anonymous requests with a rate limit error once the hourly budget is
used up, which makes composer ask for credentials and fail in containers
that have no github token. The repository entry is the only hand written
part, the requirement itself was added with the command noted below. The
test distribution used by the e2e composer tests resolves phpunit on its
own and needs no change.

runTests.sh gets an option "-j <numberOfWorkers|auto>" for the suites
unit, unitRandom and functional. It maps to the phpunit option
"--parallel" and is not put on the command line at all while unused, so
the default behaviour does not depend on a phpunit version knowing that
option. It combines with "-c": "-c" splits the suite over containers,
"-j" parallelizes the tests of one chunk inside its container.

splitFunctionalTests.php no longer discovers and counts tests with a
php-parser based heuristic, but calls "phpunit --list-tests-xml" and
reads the resulting listing. That listing carries one element per test
class including its file name, and one element per executed test with
data provider mutations counted separately. It therefore describes the
exact set of tests the very same phpunit version would run. The greedy
bin packing and the generated Build/phpunit/FunctionalTests-Job-<n>.xml
files are unchanged, so all consumers keep working. Both splitters
select the identical set of 775 test files, and the new one is byte
identical over consecutive runs, which the chunked jobs rely on since
every job runs the splitter itself. The old one counted 12438 tests
where phpunit reports 12533: seven files use "TestWith" or stack several
"DataProvider" attributes on one method, which the heuristic did not
handle. The listing costs about two seconds and needs no database.

Two defects were found in the pull request and fixed in the fork:

* Test selection was not applied to the tests executed in workers.
  "--filter", "--group" and "--exclude-group" were silently ignored for
  everything living inside a data provider, repeat or retry suite,
  because the descriptors asked the suite for its tests instead of
  iterating it, and selection in phpunit is a lazy filter iterator.
  Running one test class with "--filter" executed all 107 tests of that
  class instead of the selected 8. This hit us hard, as the whole
  database matrix builds on "--exclude-group not-<dbms>".
* Per test global and static state backup was not applied in workers, so
  state leaked between the tests of a single class. The worker rebuilds
  the test case without the configuration that TestBuilder normally
  applies, so the backup settings kept their null defaults and no
  snapshot was ever taken.

On our side, test cases needed two kinds of adjustment on top of the
change this one depends on. That change fixes unit test cases which
registered a shared scratch directory for recursive deletion instead of
the unique directory they created, so each of them removed the files of
all others on tear down. It is a defect of its own, independent of
parallel execution, and it only stayed invisible because test cases run
one after another in a single process. Parallel execution surfaced it,
which is how it was found.

* The test cases flushing the shared redis and memcached daemons are
  marked with the new attribute "DoNotRunInParallel". They wipe a daemon
  that the test runner starts once per run and shares between workers.
  Forced to overlap, they failed in three of five runs before, and pass
  in five of five now.
* The TypoScript test cases hand over token objects through their data
  providers whose "__serialize()" deliberately drops line and column to
  keep the TypoScript cache small. Those objects do not survive the
  transport into a worker, so these test cases are marked as well. Note
  this is not caused by parallel execution: the same test cases fail the
  same way with "--process-isolation" on a released phpunit.

The two pre-merge jobs exercising parallel execution are kept as
"allow_failure" for now, since this is an experiment and not a quality
gate.

Measured locally on 32 cores:

* Unit tests: 25.4s sequential, 8s with eight workers, with identical
  test and assertion counts, and green in four of four consecutive runs
  once the shared scratch directory defect was fixed.
* Functional tests fit the model well. The distribution unit of the pull
  request is one test class, which is exactly the isolation unit of the
  testing framework: it derives both the instance directory and the
  database name from the test class name, so workers do not collide. On
  a subset without database specific groups the run is 5.4 times faster
  with eight workers, with identical results, and the database server is
  not a bottleneck.
* All 775 functional test classes have a distinct instance identifier,
  so no two of them share an instance directory. Worth noting though:
  that identifier is a hash truncated to seven characters, and parallel
  execution turns a collision from harmless into two test cases
  destroying each others instance and database. A change widening it is
  prepared for the testing framework.

See https://github.com/sebastianbergmann/phpunit/pull/6784

Used command(s):
Build/Scripts/runTests.sh -s composer -- req --dev -W \
    "phpunit/phpunit:dev-parallel-test-execution-fixes as 13.4.x-dev"

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 6
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/37/95337/6 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/37/95337/6 && git cherry-pick FETCH_HEAD

Review

  • Code review CR 0 Needs a vote
  • Verified V 0 Needs a vote
    • core-ci +1 CI

Relation chain 2

  1. 95337 [WIP][TASK] Evaluate phpunit parallel test execution This change

  2. main 95338[BUGFIX] Delete only own scratch directories in unit tests This change is based on it Merged Stefan Bürk CR +2 V +2 → 14.3, 13.4 Normal +54 −23

    The chain has patch set 2 of 3, the changes above it need a rebase.

Issues 1

Files 14

Review log 612

  1. Stefan Bürk uploaded patch set 1 08/14/26, 22:10

  2. core-ci voted on patch set 1 V +1 08/14/26, 22:17 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106732

  3. Stefan Bürk updated the commit message in patch set 2 V +1 08/14/26, 22:29

  4. core-ci voted on patch set 2 V +1 08/14/26, 22:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106735

  5. Stefan Bürk uploaded patch set 3 V +1 08/14/26, 23:08

  6. Stefan Bürk uploaded patch set 4 08/14/26, 23:09

  7. core-ci voted on patch set 3 V +1 08/14/26, 23:16 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106736

  8. core-ci voted on patch set 4 V +1 08/14/26, 23:18 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106737

  9. Stefan Bürk uploaded patch set 5 V +1 08/15/26, 00:31

  10. core-ci voted on patch set 5 V +1 08/15/26, 00:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106739

  11. Stefan Bürk uploaded patch set 6 V +1 08/15/26, 12:43

  12. core-ci voted on patch set 6 V +1 08/15/26, 12:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106762