Review main Work in progress Stale

95058[WIP][TASK] Balance functional chunks by runtime

Stefan Bürk CR 0 V 0 Expert +1067 −9 updated 08/01/26, 19:22

Commit message

splitFunctionalTests.php distributes the functional test files over
the CI chunks by their number of tests. Test count predicts runtime
poorly: the cost of a functional test is dominated by fixture import
and database work, not by the number of test methods. The chunks
therefore drift apart by 5% to 12%, and a pipeline waits for the
slowest chunk of its slowest variant.

The chunks are now balanced by recorded runtime instead. The
durations are kept in Build/phpunit/FunctionalTestsTimings.json and
refreshed from the JUnit reports the functional jobs write, using
the new Build/Scripts/updateFunctionalTestsTimings.php. Test files
the timings file does not know yet, added after the last refresh,
are weighted with their number of tests times the average duration
per test. Without the file the split falls back to the previous
behaviour and balances by number of tests.

Durations are stored rounded to half seconds. The runtime of a
single test file differs by about 6% between two runs of the same
code, so a finer resolution records that noise instead of a
property of the test: at 0.01s a refresh of the file rewrites 98% of
its lines, at 0.5s it rewrites 33%, and the balance is the same
within that noise.

The committed timings are those of the slowest variant of a nightly,
functional postgres 18 php 8.5 max. The pipeline is bound by that
variant, and a split derived from it improves the faster variants as
well, while a mean over all eight variants does not: it mixes the
database profiles and loses against the current split on postgres.

Measured on two nightlies of otherwise identical code, one per split
(105808 and 105812). Test time per chunk, job overhead excluded:

  variant                       slowest chunk   deviation from
                                                   the average
  functional postgres 18 max     432s -> 411s   10.7% -> 5.5%
  functional postgres 10 min     396s -> 379s   12.2% -> 6.8%
  functional mysql 8.4 max       392s -> 369s   12.2% -> 7.1%
  functional mariadb 10.4 min    368s -> 341s   10.5% -> 5.4%
  functional mariadb 11.8 max    365s -> 362s   11.4% -> 9.8%
  functional mysql 8.0 min       403s -> 403s   12.3% -> 10.3%
  functional sqlite max          396s -> 395s    8.7% -> 8.4%
  functional sqlite min          375s -> 391s    4.9% -> 6.8%

Unlike a higher parallel count this costs nothing: same tests, same
number of jobs, no additional pipeline step, and aggregate runner
time is unchanged (22493s to 22694s, within nightly noise).

The nightly does not get shorter by this yet: it is bound by
e2e playwright composer sqlite at 472s, and the postgres chunks stay
below that either way. Pre-merge, which developers wait for, does.
Job durations of 105809 (count) against 105811 (runtime):

  functional postgres 10 pre-merge   376s -> 343s  14.2% -> 4.8%
  functional sqlite pre-merge        353s -> 338s   6.6% -> 2.7%
  functional mariadb 10.4 pre-merge  327s -> 311s  11.4% -> 7.6%

The pipeline goes from 378s to 348s.

In the nightly the two sqlite variants do not improve and sqlite min
gets worse. They run four chunks, where packing granularity is
coarse, and their profile deviates most from the postgres timings:
without a database server the fixture cost that dominates postgres
weighs far less. Timings per database would address that, at four
times the committed data.

Still open, hence WIP:

* How the timings file is refreshed and by whom. It is a manual step
  today, and it does not have to happen often, but it could also be
  automated from a nightly artifact.
* Whether ~75 KB of generated data belong into the repository, or
  whether the timings should be carried between pipelines instead.

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 2
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/58/95058/2 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/58/95058/2 && git cherry-pick FETCH_HEAD

Review

  • Code review CR 0 Needs a vote
  • Verified V 0 Needs a vote
    • core-ci +1 CI

Relation chain 3

  1. 95058 [WIP][TASK] Balance functional chunks by runtime This change

  2. main 95057[TASK] Rebalance functional test job sharding This change is based on it Merged Stefan Bürk CR +2 V +2 core-ci Easy +3 −3

    The chain has patch set 2 of 3, the changes above it need a rebase.

  3. main 94924[TASK] Skip redundant npm ci in lint suites This change is based on it Merged Stefan Bürk CR +2 V +2 core-ci Easy +3 −3

    The chain has patch set 3 of 4, the changes above it need a rebase.

Merge conflicts 1

Files 3

Review log 25

  1. Stefan Bürk uploaded patch set 1 08/01/26, 18:17

  2. core-ci voted on patch set 1 V +1 08/01/26, 18:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105811

  3. core-ci commented on patch set 1 08/01/26, 18:31 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105812

  4. Stefan Bürk uploaded patch set 2 V +1 08/01/26, 19:16

  5. core-ci voted on patch set 2 V +1 08/01/26, 19:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105813