Issue Task Resolved

#110354Rebalance functional test job sharding in CI

Stefan Bürk 08/01/26, 17:19 updated 08/06/26, 16:25

Description

Problem

The functional stage dominates both CI pipelines: it accounts for 89 percent of
the nightly job seconds, and a pipeline lasts as long as its slowest shard.

The number of shards per database variant is uniform, while the cost per variant
is not. A postgres shard runs about 1.7 times as long as a mariadb shard, yet
both variants split into 6. The two postgres variants therefore set the nightly
duration on their own, with 673s and 624s for their slowest shard, while mariadb
and sqlite finish between 382s and 401s.

Pre-merge shows the mirror image: sqlite runs 3 shards and is the critical path
at 515s, while postgres runs 10 shards and finishes at 411s.

Solution

Raise the two nightly postgres variants from 6 to 10 shards, the value pre-merge
already uses for postgres, and pre-merge sqlite from 3 to 4.

Measured on an otherwise idle CI, nightly pipeline 105802 against 105808, which
differ only by this change:

  • pipeline 679s to 483s, from more than eleven minutes to eight
  • the critical path moves off the database matrix: previously “functional
    postgres 18 max” at 673s, now the “e2e playwright composer sqlite” job at 482s
  • slowest shard of “postgres 18 max” 673s to 464s, of “postgres 10 min” 624s to
    427s

Pre-merge, pipelines 105806 and 105807 pushed within the same minute, drops from
518s to 416s. Its slowest sqlite shard goes from 515s to 413s and thereby reaches
the postgres shards at 411s, so that pipeline is balanced at its bound as well.

Cost

Sharding is not free. The overhead per job outside phpunit is only about 24s and
the per-test throughput is unchanged at 0.249s, but the phpunit runtime summed
over all shards of a variant grows from 3487s across 6 chunks to 3859s across 10
for exactly the same tests, because every additional chunk re-pays the per-chunk
warm-up. Together with the eight additional jobs the nightly aggregate rises by
6.6 percent, from 21103s to 22493s. This trades runner CPU on the shared
infrastructure for pipeline wall-clock, and is the reason the counts are not
raised further.

The counts are picked against the new bound: at 464s the slowest postgres shard
just fits below the 482s playwright job, so fewer shards would exceed it while
more would not shorten the pipeline.

Follow-up

The counts stay hand-tuned. Deriving them from recorded runtimes instead of test
counts would rebalance the remaining 100s to 150s spread within a variant without
adding chunks, and therefore without the aggregate cost. The JUnit reports the
jobs now write provide the required timings.

Reviews 1