Review main Merged

80984[FEATURE] Add CLI command to clean up missing files from database

Tung Ngo CR +2 V +2 Hard +634 −0 updated 10/23/26, 22:14

Commit message

Files deleted outside of TYPO3, e.g. via SSH or FTP, or files gone
after restoring a database dump without the matching file system,
keep their sys_file and sys_file_metadata records. The storage
indexer flags them as missing and the reports module lists them,
but nothing removes them: deleting a missing file via the FAL API
is rejected, so the only way out was to restore the file and delete
it again, or to clean the database by hand.

The new command "cleanup:missingfiles" removes the records of
missing files that are no longer referenced according to the
reference index, including soft references. Files that are still
referenced are kept and listed in verbose mode. The reference
index and the storage index of one, several or all storages can
be updated beforehand via options. Processed files are left to
"cleanup:localprocessedfiles".

The command name is taken over from the uploads/ folder command
of the same name that was removed in v12, as the File Abstraction
Layer is the only file handling left.

Review

  • Code review CR +2 Done
    • Benni Mack +2
  • Verified V +2 Done
    • Benni Mack +2
    • core-ci +1 CI

Issues 1

Files 7

Review log 3044

Show 1319 earlier messages
  1. Tung Ngo uploaded patch set 1 09/12/23, 13:10

  2. core-ci voted on patch set 1 V -1 09/12/23, 13:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46520

  3. Tung Ngo commented on patch set 2 V -1 09/12/23, 14:58

    Published edit on patch set 1.
  4. core-ci voted on patch set 2 V +1 09/12/23, 15:05 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46524

  5. Tung Ngo uploaded patch set 3 V +1 09/14/23, 10:28

  6. Tung Ngo marked it ready for review 09/14/23, 10:29

  7. core-ci voted on patch set 3 V +1 09/14/23, 10:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46572

  8. Jonas Eberle commented on patch set 3 09/15/23, 10:14

    Thank you! Just a suggestion: Maybe it makes sense to have that functionality (clean missing files from DB) as an argument to the "FAL: Update Storage Index"?
    This could be Resolves: #59312 IMHO
    v11 is in Priority Bugfix mode only Even v12 would usually not accept new features although this change is really low risk (but the Core team has to decide)

    CleanupLostFilesFromDatabase.php:148

    I think it would be nice to see some output if a sys_file.missing record is referenced and thus not deleted by this command.
  9. Tung Ngo commented on patch set 3 09/15/23, 15:39

    Reply

    I think it's a great idea. I'll try to dive into the indexer file a bit and report back.

    CleanupLostFilesFromDatabase.php:148 Reply

    I'll create a new patch for this.
  10. Sybille Peters commented on patch set 3 09/17/23, 11:20

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17

    > This console command aims to solve this problem but will not try to clean up every table (e.g., sys_file_processedfile, sys_refindex) Isn't sys_refindex not affected anyway because only file entries without references are considered?

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17

    Can you be more clear and precise about what the command actually does? Does it remove all entries in the FAL for missing files which do not have references?

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17

    IMHO, sys_file_processedfile (and processed files) should also be handled.
  11. Sybille Peters updated the commit message in patch set 4 V +1 09/17/23, 11:22

  12. core-ci voted on patch set 4 V +1 09/17/23, 11:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46630

  13. Tung Ngo uploaded patch set 5 V +1 09/18/23, 13:11

  14. core-ci voted on patch set 5 V -1 09/18/23, 13:19 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46666

  15. Tung Ngo commented on patch set 5 09/18/23, 13:28

    This change is ready for review.

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17 Reply

    IMHO, the console command cleanup:localprocessedfiles will do exactly that.

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17 Reply

    Done.

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17 Reply

    There are still sys_refindex entries left that point to the sys_file_metadata table but will be removed the next time the reference index is updated.

    CleanupLostFilesFromDatabase.php:148 Reply

    Done.
  16. Tung Ngo commented on patch set 6 V -1 09/18/23, 13:28

    Published edit on patch set 5.
  17. core-ci voted on patch set 6 V +1 09/18/23, 13:38 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46669

  18. Sybille Peters commented on patch set 6 09/18/23, 14:32

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17 Reply

    cool. Thanks for clarifying
  19. Tung Ngo commented on patch set 6 09/18/23, 16:38

    Reply

    I have a working solution, how should I proceed here? Create a completely new patch?
  1. Benni Mack commented on patch set 6 09/18/23, 16:46

    Reply

    You can amend and just push a new Patch Set here.
  2. Tung Ngo uploaded patch set 7 V +1 09/20/23, 12:44

  3. core-ci voted on patch set 7 V -1 09/20/23, 12:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46884

  4. Tung Ngo commented on patch set 8 V -1 09/20/23, 13:50

    Published edit on patch set 7.
  5. core-ci voted on patch set 8 V +1 09/20/23, 13:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46890

  6. Tung Ngo uploaded patch set 9 V +1 09/21/23, 10:07

  7. Tung Ngo marked it ready for review 09/21/23, 10:07

  8. core-ci voted on patch set 9 V +1 09/21/23, 10:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46999

  9. Tung Ngo uploaded patch set 10 V +1 09/22/23, 14:53

  10. core-ci voted on patch set 10 V -1 09/22/23, 15:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/47091

  11. Tung Ngo uploaded patch set 11 V -1 09/26/23, 12:59

  12. core-ci voted on patch set 11 V +1 09/26/23, 13:06 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/47186

  13. Tung Ngo commented on patch set 11 09/26/23, 13:10

    This change is ready for review.

    Reply

    Done

    Reply

    Done
  14. Tung Ngo marked it work in progress 10/10/23, 10:42

  15. Tung Ngo commented on patch set 11 10/10/23, 10:43

    This change is ready for review.

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17 Reply

    After further testing, cleanup:localprocessedfiles does not clean up sys_file_processedfile (and processed files) for missing files. I'll try to add the feature. The “Remove Temporary Assets” maintenance module could be costly here (for large instances) if all files in _processed_files are removed as TYPO3 has to recreate them.
  16. Tung Ngo marked it work in progress 10/10/23, 10:43

  17. Benni Mack uploaded patch set 12 V +1 10/23/26, 15:01

  18. Benni Mack uploaded patch set 13 10/23/26, 15:11

  19. Benni Mack commented on patch set 13 10/23/26, 15:11

    This change is ready for review.

    Feature-59312-ConsoleCommandToCleanUpIndexedMissingFilesWithoutReferencesFromDatabase.rst:17 Reply

    Done
  20. core-ci voted on patch set 12 V -1 10/23/26, 15:53 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112338

  21. core-ci voted on patch set 13 V -1 10/23/26, 16:25 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112340

  22. Benni Mack rebased onto patch set 14 V -1 10/23/26, 21:35

  23. Benni Mack voted on patch set 14 CR +2 V +2 10/23/26, 21:37

  24. core-ci voted on patch set 14 V +1 10/23/26, 21:55 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112370

  25. Benni Mack merged patch set 15 10/23/26, 22:14