Review main Abandoned

88102[BUGFIX] Prevent accidental deletion in lowlevel

Sybille Peters CR 0 V 0 Normal +96 −24 updated 02/17/25, 19:06

Commit message

The command cleanup:orphanrecords in EXT:lowlevel could have deleted
non-orphan records (e.g. pages) if they were created after the command
started.

This is due to the nature how the command collects the possible orphans:
First, it creates a list of connected records. If a record is created
after the command started, it will not be included in this list.

We now check the timestamp of the record. This means, only tables
with timestamps can be considered.

Additionally, we also exclude the following tables (where orphan
checking makes no sense):

- tables sys_log and sys_history
- tables with no TCA ctrl array
- tables where ctrl.rootLevel is set to 1 (these can only exist on
  the rootlevel page with uid=0)

The related issue contains steps to reproduce.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 1

Review log 1112

  1. Sybille Peters uploaded patch set 1 02/08/25, 13:12

  2. core-ci voted on patch set 1 V +1 02/08/25, 13:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/73804

  3. Stefan Bürk voted on patch set 1 CR -1 02/08/25, 21:40

  4. Christian Kuhn commented on patch set 1 02/10/25, 12:11

    well. the main issue with cleanup:orphanrecords and cleanup:missingrelations is that they are both insanely slow. this is roughly due to this strategy: get a page, do x queries per page for *all* existing tables if there are some records. compile all that into an humongous array to then work on it, easily leading to hundreds of megabytes memory. with my "not *that* huge" local instance, a run already takes 10 minutes, and if i unload ext:styleguide (which brings many tables), it is *still* 3 minutes. these commands are simply unusable for any not trivially small instance. i fixed a similar strategy with cleanup:flexforms in v13 and the result was stunning: from many minutes to a couple of seconds with nearly no memory. codesize shrinked to 1/3. as such, i wonder if it is really useful to hack on these commands and adding patches that add quite a bit of code. we are probably much better off refactoring the codebase to a sane solution first, to then see what still needs to be done. as such, i'm not a fan of the patch at the moment, and would rather try to pick it up after a general modernization of the two commands - i think it will be much more easy to solve it afterwards and probably won't add additional runtime overhead again. note these commands have *never* been looked at or touched in details since 3.8 or so. they are rotting around for a very long time and nobody ever cared to work on them. i wonder how many instances actively use them. what do you think?
  5. Sybille Peters commented on patch set 1 02/11/25, 07:27

    Reply

    You are right, no doubt (about the inefficiency, memory, etc.). My approach was: There is a really nasty bug. I mean this command deleting non-orphaned pages. And doing this fix took me less than 30 minutes. It may be better to completely overhaul cleanup, but who is willing to do it and when? My main goal is to fix this nasty bug ASAP (issue has been open 4 years). (we could reduce amount of changed code by reverting $io => $this->io, if this would help). So, is this overhaul happening any time soon? And wouldn't it be better to move "dbdoctor" to core and add it there? Or would it be an alternative to add a warning somewhere (the docs tell you to backup. but they don't tell you to lock the Backend before running the command).
  6. Sybille Peters commented on patch set 1 02/11/25, 09:11

    Additional comment: if this has no chance of getting merged, let's abandon and not spend additional time on it.
  7. Christian Kuhn commented on patch set 1 02/11/25, 09:55

    Reply

    Let's keep it pending for a while - I'm kinda certain we'll refactor cleanup:orphanrecords quite soon, and depending on its implementation we can see wether this change still makes sense.

    Reply

    Well, dbdoctor will not be merged to core as such - there are too many loose ends (i hope to get a v13 compat version ready, soon, though). But i looked at dbdoctor for cleanup:orphanrecords already: dbdoctor has two health checks that - combined - do what cleanup:orphanrecords does. It shouldn't be *that* hard to grab the dbdoctor code as basis and create a refactored cleanup:orphanrecords from it. It's probably a few hours to do, and I was close to hack on that already, but got lost in details: I for instance thought about dropping the DH call that does the deletion in favor of a direct hard delete, but was not 100% sure on it, yet.
  8. Sybille Peters commented on patch set 1 02/11/25, 17:54

    Reply

    ok
  9. Christian Kuhn commented on patch set 1 02/13/25, 23:14

    Reply

    hey. I quickly hacked up a PoC/WIP change: https://review.typo3.org/c/Packages/TYPO3.CMS/+/88213 This creates a list of valid pages without creating a full tree. One DB query per TCA table, minimal memory impact. Runtime before: >10 minutes, after: 0.3 seconds. Could you maybe play around with this a bit to see if it finds the same data set that the old implementation comes up with?
  10. Christian Kuhn commented on patch set 1 02/17/25, 19:04

    OrphanRecordsCommand.php:166

    i'm skipping this in new implementation: these tables have no TCA.
  11. Christian Kuhn commented on patch set 1 02/17/25, 19:05

    Reply

    i'll abandon here now: i'm about to push a solution with https://review.typo3.org/c/Packages/TYPO3.CMS/+/88213.

    Reply

    Done
  12. Christian Kuhn abandoned the change 02/17/25, 19:06