80984[FEATURE] Add CLI command to clean up missing files from database
Commit message
Review
-
Code review CR +2 Done
- Benni Mack +2
-
Verified V +2 Done
- Benni Mack +2
- core-ci +1 CI
Issues 1
Files 7
- typo3/sysext/core/Documentation/Changelog/15.0/Feature-59312-CleanupCommandForMissingFiles.rstnew+72 −0
- typo3/sysext/lowlevel/Classes/Command/MissingFilesCommand.phpnew+355 −0
- typo3/sysext/lowlevel/Tests/Functional/Command/Fixtures/MissingFilesCommandDryRunResult.csvnew+26 −0
- typo3/sysext/lowlevel/Tests/Functional/Command/Fixtures/MissingFilesCommandImport.csvnew+33 −0
- typo3/sysext/lowlevel/Tests/Functional/Command/Fixtures/MissingFilesCommandResult.csvnew+21 −0
- typo3/sysext/lowlevel/Tests/Functional/Command/CliCommandTest.php+1 −0
- typo3/sysext/lowlevel/Tests/Functional/Command/MissingFilesCommandTest.phpnew+126 −0
Review log 3044
Show 1319 earlier messages
-
Tung Ngo uploaded patch set 1 09/12/23, 13:10
-
core-ci voted on patch set 1 V -1 09/12/23, 13:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46520
-
Tung Ngo commented on patch set 2
V -109/12/23, 14:58Published edit on patch set 1. -
core-ci voted on patch set 2 V +1 09/12/23, 15:05 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46524
-
Tung Ngo uploaded patch set 3
V +109/14/23, 10:28 -
Tung Ngo marked it ready for review 09/14/23, 10:29
-
core-ci voted on patch set 3 V +1 09/14/23, 10:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46572
-
Jonas Eberle commented on patch set 3 09/15/23, 10:14
Thank you! Just a suggestion: Maybe it makes sense to have that functionality (clean missing files from DB) as an argument to the "FAL: Update Storage Index"?This could be Resolves: #59312 IMHOv11 is in Priority Bugfix mode only Even v12 would usually not accept new features although this change is really low risk (but the Core team has to decide)CleanupLostFilesFromDatabase.php:148I think it would be nice to see some output if a sys_file.missing record is referenced and thus not deleted by this command. -
Tung Ngo commented on patch set 3 09/15/23, 15:39
Reply
I think it's a great idea. I'll try to dive into the indexer file a bit and report back.CleanupLostFilesFromDatabase.php:148ReplyI'll create a new patch for this. -
Sybille Peters commented on patch set 3 09/17/23, 11:20
Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17> This console command aims to solve this problem but will not try to clean up every table (e.g., sys_file_processedfile, sys_refindex) Isn't sys_refindex not affected anyway because only file entries without references are considered?Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17Can you be more clear and precise about what the command actually does? Does it remove all entries in the FAL for missing files which do not have references?Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17IMHO, sys_file_processedfile (and processed files) should also be handled. -
Sybille Peters updated the commit message in patch set 4
V +109/17/23, 11:22 -
core-ci voted on patch set 4 V +1 09/17/23, 11:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46630
-
Tung Ngo uploaded patch set 5
V +109/18/23, 13:11 -
core-ci voted on patch set 5 V -1 09/18/23, 13:19 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46666
-
Tung Ngo commented on patch set 5 09/18/23, 13:28
This change is ready for review.Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17ReplyIMHO, the console command cleanup:localprocessedfiles will do exactly that.Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17ReplyDone.Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17ReplyThere are still sys_refindex entries left that point to the sys_file_metadata table but will be removed the next time the reference index is updated.CleanupLostFilesFromDatabase.php:148ReplyDone. -
Tung Ngo commented on patch set 6
V -109/18/23, 13:28Published edit on patch set 5. -
core-ci voted on patch set 6 V +1 09/18/23, 13:38 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46669
-
Sybille Peters commented on patch set 6 09/18/23, 14:32
Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17Replycool. Thanks for clarifying -
Tung Ngo commented on patch set 6 09/18/23, 16:38
Reply
I have a working solution, how should I proceed here? Create a completely new patch?
-
Benni Mack commented on patch set 6 09/18/23, 16:46
Reply
You can amend and just push a new Patch Set here. -
Tung Ngo uploaded patch set 7
V +109/20/23, 12:44 -
core-ci voted on patch set 7 V -1 09/20/23, 12:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46884
-
Tung Ngo commented on patch set 8
V -109/20/23, 13:50Published edit on patch set 7. -
core-ci voted on patch set 8 V +1 09/20/23, 13:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46890
-
Tung Ngo uploaded patch set 9
V +109/21/23, 10:07 -
Tung Ngo marked it ready for review 09/21/23, 10:07
-
core-ci voted on patch set 9 V +1 09/21/23, 10:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46999
-
Tung Ngo uploaded patch set 10
V +109/22/23, 14:53 -
core-ci voted on patch set 10 V -1 09/22/23, 15:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/47091
-
Tung Ngo uploaded patch set 11
V -109/26/23, 12:59 -
core-ci voted on patch set 11 V +1 09/26/23, 13:06 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/47186
-
Tung Ngo commented on patch set 11 09/26/23, 13:10
This change is ready for review.Reply
DoneReply
Done -
Tung Ngo marked it work in progress 10/10/23, 10:42
-
Tung Ngo commented on patch set 11 10/10/23, 10:43
This change is ready for review.Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17ReplyAfter further testing, cleanup:localprocessedfiles does not clean up sys_file_processedfile (and processed files) for missing files. I'll try to add the feature. The “Remove Temporary Assets” maintenance module could be costly here (for large instances) if all files in _processed_files are removed as TYPO3 has to recreate them. -
Tung Ngo marked it work in progress 10/10/23, 10:43
-
Benni Mack uploaded patch set 12
V +110/23/26, 15:01 -
Benni Mack uploaded patch set 13 10/23/26, 15:11
-
Benni Mack commented on patch set 13 10/23/26, 15:11
This change is ready for review.Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17ReplyDone -
core-ci voted on patch set 12 V -1 10/23/26, 15:53 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112338
-
core-ci voted on patch set 13 V -1 10/23/26, 16:25 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112340
-
Benni Mack rebased onto patch set 14
V -110/23/26, 21:35 -
Benni Mack voted on patch set 14 CR +2 V +2 10/23/26, 21:37
-
core-ci voted on patch set 14 V +1 10/23/26, 21:55 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112370
-
Benni Mack merged patch set 15 10/23/26, 22:14