Review 13.4 Merged

89160[BUGFIX] Restore nested `CompositeExpression` performance

Stefan Bürk CR +2 V +2 Normal +183 −22 updated 04/17/25, 15:59

Commit message

Upgrading to Doctrine DBAL 4 for TYPO3 v13 required to adopt
quite some changes to decorated or extended classes like the
`CompositeExpression` class which has changed to `immutable`
class design. The TYPO3 implementation had to be aligned and
ensuring known normalization like properly skipping empty or
single item parts properly.

The refactoring in `CompositeExpression` for #102875 added a
performance penalty to the system, which mainly boils down to
two things:

* `array_filter()` returns the filtered array as return value
  and does not operate on the passed array like `array_walk`,
  and the return value has not been used in methods `or()`,
  `and()`, `with()` and `isEmptyPart()` doing the filtering
  again and again, ending up doing it only within the method
  `__toString()` at a late point and called multiple times.
* Static `CompositeExpression::isEmptyPart()` called nearly
  every time the `__toString()` through the string cast and
  ending in recursion which contributed to the performance
  penalty.

To mitigate the performance penalty and return to comparable
performance of TYPO3 v12 following modification are done:

* Use `array_filter()` return value for further processing
  in method `CompositeExpression::with()`
* Remove obsolete filtering in `CompositeExpression::and()`
  and `CompositeExpression::or()`, both calling `with()`.
* Remove part filtering in `__toString()`, which has been
  now already done in `with()` and no need to do it again.
* Streamlined `CompositeExpression::isEmptyPart()` making
  cheap checks first, respect the fact that the TYPO3 own
  implementation is reliable to use only the count check
  and doing extensive trim check only for Doctrine DBAL
  `CompositeExpression` objects.

To make some measurements, following CLI command has been
used:

  <?php
  declare(strict_types=1);

  namespace SBUERK\TestDummy\Command;

  use Symfony\Component\Console\Attribute\AsCommand;
  use Symfony\Component\Console\Command\Command;
  use Symfony\Component\Console\Input\InputInterface;
  use Symfony\Component\Console\Output\OutputInterface;
  use Symfony\Component\Stopwatch\Stopwatch;
  use TYPO3\CMS\Core\Database\Query\Expression\CompositeExpression;

  #[AsCommand(name: 'sbuerk:composite-expression-test')]
  final class CompositeExpressionTestCommand extends Command
  {
    private int $created = 0;
    private const MAX_LEVEL = 4;
    private const NUM_PARTS_TO_CREATE = 30;

    protected function execute(
      InputInterface $input,
      OutputInterface $output
    ): int {
      $watch = new Stopwatch(true);

      $event = $watch->start('create');
      $compositeExpression = $this->createCompositeExpression();
      $elapsed = (string)$event->stop();

      $output->writeln('Elapsed: ' .  . ' CREATED: ' . $this->created);

      return Command::SUCCESS;
    }

    private function createCompositeExpression(
      int $level = 0
    ): CompositeExpression {
      if ($level > self::MAX_LEVEL) {
        $this->created++;
        return CompositeExpression::or(...['dummy = 1']);
      }
      $nextLevel = $level + 1;
      $parts = [];
      for ($i=0; $i < self::NUM_PARTS_TO_CREATE; $i++) {
        $parts[] = $this->createCompositeExpression($nextLevel);
        $this->created++;
      }
      return CompositeExpression::or(...$parts);
    }
  }

And here some stats. Note, that the outcome is literally
unrealistic and having the need to generate that much of
CompositeExpression other issues will raise up, but the
extreme outcome makes the difference clear.

Baseline - TYPO3 12.4-dev
-------------------------

| MAX_LEVEL | NUM_CREATE | TIME       | CREATED  |
|-----------|------------|------------|----------|
| 2         | 30         | 82 ms      | 54930    |
| 3         | 30         | 2492 ms    | 1647930  |
| 4         | 30         | 82346 ms   | 49437930 |

main - current
--------------

| MAX_LEVEL | NUM_CREATE | TIME       | CREATED |
|-----------|------------|------------|---------|
| 2         | 30         | 369 ms     | 54930   |
| 3         | 30         | 22991 ms   | 1647930 |
| 4         | 30         | ~ aborted  | 0       |

this change
-----------

| MAX_LEVEL | NUM_CREATE | TIME       | CREATED  |
|-----------|------------|------------|----------|
| 2         | 30         | 38 ms      | 54930    |
| 3         | 30         | 1147 ms    | 1647930  |
| 4         | 30         | 41568 ms   | 49437930 |

Review

  • Code review CR +2 Done
    • Stefan Bürk +2
  • Verified V +2 Done
    • Stefan Bürk +2
    • core-ci +1 CI

Issues 2

Files 2

Review log 35

  1. Stefan Bürk commented on patch set 1 04/17/25, 15:52

    Cherry Picked from branch main.
  2. core-ci voted on patch set 1 V -1 04/17/25, 15:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77899

  3. core-ci voted on patch set 1 V +1 04/17/25, 15:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77899

  4. Stefan Bürk voted on patch set 1 CR +2 V +2 04/17/25, 15:59

  5. Stefan Bürk merged patch set 2 04/17/25, 15:59