Issue Bug Closed

#102050SOLVED: Version 12.4.6 increased our server load tenfold

Leonie Krenzer 09/27/23, 13:40 updated 09/29/23, 07:53

Description

UPDATE
We rolled out version 12.4.6 again this morning, including the latest updates to the third party extensions we use as well as our own extensions. The problem did not occur again. As the core version hasn’t changed in the meantime it seems safe to assume that this was not an issue with the 12.4.6 version after all. Sorry for the confusion caused, we were very confused ourselves and the core update seemed as the most likely candidate at the time.

——————- original description ——————

We updated our product from 12.4.5 to 12.4.6 today. As a result of the update, the CPU and network traffic load on our server increased by about a factor 10 (see attached Grafana screenshots*). In htop, we were able to see a large number of php-fpm child processes with a runtime of up to four minutes. php-fpm also reguarly logged timeouts in seemingly random files, as well as warnings for too many child processes being created. The sys_log table didn’t log any irregularities.

We are hosting TYPO3 on a 22.04.3 Ubuntu system, using NGINX and php8.1.

Honestly, I have no clue why it broke but I’ll be happy to provide more information about the scenario. We didn’t notice this at all on our staging environment, this seems to have been exacerbated by actual user traffic.

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.

Files 2

History 4

  1. Leonie Krenzer commented 09/28/23, 07:03

    I noticed my image descriptions were a little imprecise: We decreased the version to 12.4.5 at about 2:30 PM, at 2:55 we published the changes to our own extensions again, but this time with an updated composer.lock to keep the system on version 12.4.5 while still including our changes – so we were able to rule out immediate side effects from our own changes.

  2. Leonie Krenzer changed subject Version 12.4.6 increased our server load tenfold SOLVED: Version 12.4.6 increased our server load tenfold description 09/29/23, 07:26

  3. Leonie Krenzer changed description 09/29/23, 07:27

  4. Stefan Bürk commented and changed status New Closed 09/29/23, 07:53

    I was in contact with Leonie over slack direct messages. This isse could
    not narrowed down to any core commits, and also first conclusions related
    to 3rd party extensions did not showed up again.

    The upgraded project has been re-deployed, without having the raised values
    in the monitoring. So it could be that 3rd party extension issue has been
    fixed as new versions has been used partly for the re-deployment.

    For now, there are now hints that this is core issue, and as no other reports
    pointing to a similar isse has been reported I’m going to close this issue now.

    If something changes feel to ask for reopening / reopen it.