Review main Merged

61654[BUGFIX] Make wizard SeparateSysHistoryFromSysLogUpdate repeatable

Jigal van Hemert CR +2 V +2 Hard +273 −72 updated 03/07/20, 10:00

Commit message

To prevent memory issues with huge sys_history tables and to prevent
timeouts when handling large amounts of data the wizard processes small
batches of records at a time, keeps track of its progress in the registry
and this way allows repeated runs without losing data.
If both the registry and sys_history share the same connection a database
transaction is used to make sure that modifying data and progress
tracking are in sync.

Review

  • Code review CR +2 Done
    • Susi Moog +2
    • Benni Mack +1
  • Verified V +2 Done
    • Susi Moog +2
    • Roland Golla +1
    • TYPO3com +1 CI

Issues 1

Files 1

Review log 1520

  1. Jigal van Hemert uploaded patch set 1 09/10/19, 07:07

  2. TYPO3com voted on patch set 1 V -1 09/10/19, 07:07 Completed build in 11s on Tue, 10 Sep, 07:07 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36237

  3. Andreas Kienast uploaded patch set 2 09/10/19, 07:12

  4. TYPO3com voted on patch set 2 V +1 09/10/19, 07:23 Completed build in 10m 50s on Tue, 10 Sep, 07:23 AM Test Summary: 47398 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36238

  5. Benni Mack voted on patch set 2 CR +1 09/11/19, 06:06

  6. Susi Moog commented on patch set 2 09/11/19, 10:58

    SeparateSysHistoryFromSysLogUpdate.php:28

    just from the commit message: should this also implement RepeatableInterface?
  7. Jigal van Hemert commented on patch set 2 09/13/19, 19:04

    > Patch Set 2: > > (1 comment) (1 reply)

    SeparateSysHistoryFromSysLogUpdate.php:28 Reply

    You're absolutely right. It was lost between the version I made to perform an upgrade to v9 and the version in current master. I guess that I just forgot to copy the class line.
  8. Jigal van Hemert commented on patch set 3 09/13/19, 19:09

    Published edit on patch set 2.
  9. Jigal van Hemert commented on patch set 3 09/13/19, 19:11

    Added RepeatableInterface

    SeparateSysHistoryFromSysLogUpdate.php:28 Reply

    Done
  10. TYPO3com voted on patch set 3 V +1 09/13/19, 19:19 Completed build in 09m 58s on Fri, 13 Sep, 07:19 PM Test Summary: 47556 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36440

  11. Susi Moog rebased onto patch set 4 11/06/19, 10:01

  12. TYPO3com voted on patch set 4 V +1 11/06/19, 10:36 Completed build in 59m 36s on Wed, 6 Nov, 10:36 AM Test Summary: 48131 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-38033

  13. Alexander Schnitzler commented on patch set 4 11/07/19, 13:47

    SeparateSysHistoryFromSysLogUpdate.php:31

    btw: just another idea to reduce the footprint. If we first fetch all the uid's and then let a generator fetch single rows from the database for each record, it's more queries but less memory. I'm fine with your proposed solution here but I guess we could also solve the memory issues without using batches. Batches are a good idea though due to the php timeout when being executed in the browser and not on cli.

    SeparateSysHistoryFromSysLogUpdate.php:32

    let's declare visibility here. I guess you want to have it private, right?

    SeparateSysHistoryFromSysLogUpdate.php:117

    It's just an update wizard but can we use constants here that explain what phase 0 and 1 stand for?

    SeparateSysHistoryFromSysLogUpdate.php:239

    The logic of phase 0 and 1 are independent from each other, right? If so, can't we put the code in dedicated methods just to decrease the cyclic complexity and have less indentations?
  14. Jigal van Hemert uploaded patch set 5 11/13/19, 23:33

  15. Jigal van Hemert commented on patch set 5 11/13/19, 23:35

    SeparateSysHistoryFromSysLogUpdate.php:31 Reply Unresolved

    The current solution keeps track of the progress and allows you to start the wizard again if it stopped for some reason and it will continue from where it stopped. The memory usage is pretty constant (depending on the batch size) while collecting all UIDs will again store information on all records in memory. The amount stored for each record is very small but I looked for a solution that would not depend on the total number of records.

    SeparateSysHistoryFromSysLogUpdate.php:32 Reply

    Done

    SeparateSysHistoryFromSysLogUpdate.php:117 Reply

    Done

    SeparateSysHistoryFromSysLogUpdate.php:239 Reply

    Done
  16. TYPO3com voted on patch set 5 V +1 11/13/19, 23:40 Completed build in 07m 09s on Wed, 13 Nov, 11:40 PM Test Summary: 48320 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-38355

  17. Benni Mack voted on patch set 5 CR +1 03/02/20, 21:42

  18. Roland Golla commented on patch set 5 V +1 03/06/20, 16:30

    Upgrade wizard is working fine for me. But I do not now if I tested everything right ;)
  19. Susi Moog voted on patch set 5 V +2 CR +2 03/07/20, 09:57

  20. Susi Moog merged patch set 6 03/07/20, 10:00