58721[FEATURE] Introduce command to clean up processed files
Commit message
Review
-
Code review CR +2 Done
- Stefan Bürk +2
- Benni Mack +1
-
Verified V +2 Done
- Stefan Bürk +2
- Benni Mack +1
- core-ci +1 CI
Issues 1
Files 13
- typo3/sysext/core/Documentation/Changelog/12.3/Feature-45039-CommandToCleanUpLocalProcessedFiles.rst2new+49 −0
- typo3/sysext/core/Tests/Acceptance/Application/Cli/CommandCest.php2+1 −0
- typo3/sysext/lowlevel/Classes/Command/CleanUpLocalProcessedFilesCommand.php10new+161 −0
- typo3/sysext/lowlevel/Classes/Service/CleanUpLocalProcessedFilesService.php12new+212 −0
- typo3/sysext/lowlevel/Configuration/Services.yaml+6 −0
- typo3/sysext/lowlevel/Tests/Functional/Clean/CleanUpLocalProcessedFilesTest.php10new+168 −0
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/DataSet/FileWithoutProcessedFileRecord.pngnewbinary
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/DataSet/FileWithoutProcessedFileRecord2.pngnewbinary
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/DataSet/image.pngnewbinary
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/DataSet/NotReferencedImage.pngnewbinary
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/DataSet/NotReferencedImage2.pngnewbinary
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/DataSet/sys_file_processedfile.csv4new+4 −0
- typo3/sysext/lowlevel/Tests/Functional/Fixtures/Modify/oneDeleted.csv2new+3 −0
Review log 117183
Show 98158 earlier messages
-
Eike Starkmann uploaded patch set 1 10/26/18, 15:13
-
TYPO3com voted on patch set 1 V -1 10/26/18, 15:33 Completed build in 35s on Fri, 26 Oct, 05:33 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29619
-
Eike Starkmann uploaded patch set 2 10/26/18, 16:04
-
TYPO3com voted on patch set 2 V +1 10/26/18, 16:15 Completed build in 11m 21s on Fri, 26 Oct, 06:15 PM Test Summary: 37819 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29621
-
Joerg Kummer commented on patch set 2 10/27/18, 09:18
-
Eike Starkmann uploaded patch set 3 10/27/18, 16:42
-
TYPO3com voted on patch set 3 V -1 10/27/18, 16:42 Completed build in 09s on Sat, 27 Oct, 06:42 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29686
-
Eike Starkmann uploaded patch set 4 10/27/18, 21:02
-
TYPO3com voted on patch set 4 V -1 10/27/18, 21:02 Completed build in 12s on Sat, 27 Oct, 11:02 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29704
-
Eike Starkmann updated the commit message in patch set 5 10/27/18, 21:03
-
TYPO3com voted on patch set 5 V -1 10/27/18, 21:03 Completed build in 16s on Sat, 27 Oct, 11:03 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29705
-
Eike Starkmann rebased onto patch set 6 10/27/18, 21:14
-
TYPO3com commented on patch set 6 10/27/18, 21:14 Completed build in 09s on Sat, 27 Oct, 11:14 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29707
-
Eike Starkmann uploaded patch set 7 10/27/18, 21:16
-
TYPO3com voted on patch set 7 V -1 10/27/18, 21:28 Completed build in 11m 18s on Sat, 27 Oct, 11:28 PM Test Summary: 1 of 38206 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29708
-
Eike Starkmann uploaded patch set 8 10/27/18, 21:49
-
TYPO3com voted on patch set 8 V +1 10/27/18, 22:01 Completed build in 11m 34s on Sun, 28 Oct, 12:01 AM Test Summary: 37846 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29713
-
Eike Starkmann uploaded patch set 9 10/28/18, 09:17
-
Eike Starkmann uploaded patch set 10 10/28/18, 09:27
-
TYPO3com voted on patch set 9 V +1 10/28/18, 09:30 Completed build in 13m 10s on Sun, 28 Oct, 10:30 AM Test Summary: 37846 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29715
-
TYPO3com voted on patch set 10 V +1 10/28/18, 09:40 Completed build in 13m 05s on Sun, 28 Oct, 10:40 AM Test Summary: 37846 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29717
-
Eike Starkmann uploaded patch set 11 10/28/18, 10:04
-
TYPO3com voted on patch set 11 V +1 10/28/18, 10:18 Completed build in 14m 12s on Sun, 28 Oct, 11:18 AM Test Summary: 37846 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29722
-
Ursula Klinger commented on patch set 11 CR +1 10/28/18, 10:19
I do not know really good FAL, so I cannot verify the method cleanFiles, but the code looks good to me. -
Eike Starkmann uploaded patch set 12 10/28/18, 10:41
-
TYPO3com voted on patch set 12 V -1 10/28/18, 10:41 Completed build in 10s on Sun, 28 Oct, 11:41 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29726
-
Eike Starkmann uploaded patch set 13 10/28/18, 11:07
-
TYPO3com voted on patch set 13 V -1 10/28/18, 11:26 Completed build in 18m 58s on Sun, 28 Oct, 12:26 PM Test Summary: 3 of 38209 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29734
-
Eike Starkmann uploaded patch set 14 10/28/18, 11:54
-
Eike Starkmann uploaded patch set 15 10/28/18, 12:05
-
TYPO3com voted on patch set 14 V -1 10/28/18, 12:06 Completed build in 11m 42s on Sun, 28 Oct, 01:06 PM Test Summary: 3 of 38209 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29741
-
TYPO3com voted on patch set 15 V +1 10/28/18, 12:17 Completed build in 11m 28s on Sun, 28 Oct, 01:17 PM Test Summary: 37849 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29742
-
Eike Starkmann uploaded patch set 16 10/28/18, 12:25
-
TYPO3com voted on patch set 16 V +1 10/28/18, 12:44 Completed build in 16m 39s on Sun, 28 Oct, 01:44 PM Test Summary: 37849 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29746
-
Eike Starkmann uploaded patch set 17 10/28/18, 12:50
-
TYPO3com voted on patch set 17 V +1 10/28/18, 13:23 Completed build in 22m 48s on Sun, 28 Oct, 02:23 PM Test Summary: 37849 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-29757
-
Jonas Eberle commented on patch set 17 10/29/18, 16:14
Tried to test with the introduction package. 1) checked references to a file in "filelist" module 2) rendered the frontend, remembered processed file 3) Removed the image reference from the content element 4) checked that "filelist" module now does not show the reference any more 5) called bin/typo3 cleanup:processedfiles --dry-run --verbose -> got no output (would expect at least something due to --verbose) called bin/typo3 cleanup:processedfiles -> (in line with --dry-run, but I would expect that processed file to be deleted) 6) checked for processed file -> still there 7a) deleted image via "filelist" module -> processed file gets deleted (by Typo3) 7b) I tried another approach: I deleted a referenced file outside of Typo3. -> still bin/typo3 cleanup:processedfiles does not delete the processed file Can you provide some steps to test?Shouldn't this feature be rather called "Typo3 CLI command to..." Because as far as I can see this is not a scheduler task but a symfony CLI command to be used with ./bin/typo3Feature-58721-Scheduler task to clean up unused processed images.rst:4Rather "clean up command" or "Typo3 CLI command to clean up..." than scheduler taskCleanProcessedFilesCommand.php:126I would expect some verbose output here with --verbose, e.g. currently checked folders and files -
Eike Starkmann rebased onto patch set 18 11/06/18, 14:19
-
TYPO3com voted on patch set 18 V +1 11/06/18, 14:29 Completed build in 09m 35s on Tue, 6 Nov, 03:29 PM Test Summary: 48470 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-30035
-
Eike Starkmann uploaded patch set 19 12/04/18, 15:20
-
TYPO3com voted on patch set 19 V +1 12/04/18, 15:31 Completed build in 10m 16s on Tue, 4 Dec, 04:31 PM Test Summary: 48557 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-30439
-
Josef Glatz commented on patch set 17 12/05/18, 06:22
Reply
@Jonas: A Symfony Console Command is available as scheduler task per default. You can disable it from showing up within ext:scheduler in TYPO3 9.5 LTS. This info can be found in one of the ChangeLog RsT's -
Jonas Eberle wrote 06/28/19, 18:01
Removed reviewer Jonas Eberle. -
Benni Mack uploaded patch set 20 02/14/20, 20:24
-
Georg Ringer updated the commit message in patch set 21 02/14/20, 20:28
-
TYPO3com voted on patch set 20 V +1 02/14/20, 21:31 Completed build in 1h 03m 24s on Fri, 14 Feb, 09:31 PM Test Summary: 71262 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-89
-
TYPO3com voted on patch set 21 V +1 02/14/20, 21:44 Completed build in 57m 26s on Fri, 14 Feb, 09:44 PM Test Summary: 71268 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-92
-
Stefan Froemken commented on patch set 21 CR -1 02/15/20, 09:00
CleanProcessedFilesCommand.php:58Only a nice to have. Can you move that line to top or below the option initialization, to have them more grouped?CleanProcessedFilesCommand.php:75Add empty line between description and parameters. And add bool typeCleanProcessedFilesCommand.php:78add bool typeCleanProcessedFilesCommand.php:90Please move the initialization out of the foreach.CleanProcessedFilesCommand.php:94Only a nice to have: Use sprintf('Would... identifier "%s" and UID %d', $identifier, $uid) instead of this concatenation with different kinds of datatypes.CleanProcessedFilesCommand.php:99Don't use uid as string in DB Query.CleanProcessedFilesCommand.php:110boolCleanProcessedFilesCommand.php:113Add bool typeCleanProcessedFilesCommand.php:120In rst file you wrote "only works for local drivers". But here you get all storages.CleanProcessedFilesTest.php:3remove empty lineCleanProcessedFilesTest.php:74add space -
TYPO3com voted on patch set 21 V -1 04/06/20, 04:23 Completed build in 19m 11s on Mon, 6 Apr, 04:23 AM Test Summary: 9 of 73859 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2208
-
Benni Mack commented on patch set 21 09/07/20, 18:59
ping. is anyone going to pick this up? Otherwise I'd suggest to abandon this issue until somebody is working on it again. -
Jörg Bösche commented on patch set 21 09/09/20, 08:52
CleanProcessedFilesCommand.php:58ReplyDoneCleanProcessedFilesCommand.php:75ReplyDoneCleanProcessedFilesCommand.php:78ReplyDoneCleanProcessedFilesCommand.php:90ReplyDoneCleanProcessedFilesCommand.php:94ReplyDoneCleanProcessedFilesCommand.php:99ReplyDoneCleanProcessedFilesCommand.php:110ReplyDoneCleanProcessedFilesCommand.php:113ReplyDoneCleanProcessedFilesCommand.php:120ReplyDoneCleanProcessedFilesTest.php:3ReplyDoneCleanProcessedFilesTest.php:74ReplyDone -
Jörg Bösche uploaded patch set 22 09/09/20, 08:52
-
TYPO3com voted on patch set 22 V +1 09/09/20, 09:03 Completed build in 11m 29s on Wed, 9 Sep, 09:03 AM Test Summary: 73182 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5389
-
Benni Mack updated the commit message in patch set 23 09/23/20, 14:15
-
TYPO3com voted on patch set 23 V +1 09/23/20, 14:43 Completed build in 23m 20s on Wed, 23 Sep, 02:43 PM Test Summary: 73691 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5867
-
Guido Schmechel commented on patch set 23 09/23/20, 15:42
CleanProcessedFilesCommand.php:34I would suggest that we only have the DryRun option. The other options are confusing in my opinion.CleanProcessedFilesCommand.php:75Is this option safe? Don't the DB entries and files have to be deleted together, if possible? If I understand it correctly, one also deletes the DB entry in the cleanFiles().CleanProcessedFilesCommand.php:97Tablenam as constant. We have several locationsCleanProcessedFilesCommand.php:115Don't we also have to consider the type of storage?CleanProcessedFilesCommand.php:119Double "identifier"CleanProcessedFilesCommand.php:153Maybe a constant makes more sense? Otherwise the variable might get lost during refactoring.CleanProcessedFilesCommand.php:157Maybe we can use code from the install tool? $repository = GeneralUtility::makeInstance(ProcessedFileRepository::class); return $repository->removeAll($storageUid); This way we prevent duplicate code. Only that we could not use the DryRun here. -
Guido Schmechel voted on patch set 23 CR -1 09/23/20, 15:42
-
Stefan Froemken uploaded patch set 24 09/30/20, 21:58
-
Stefan Froemken commented on patch set 24 09/30/20, 22:02
CleanProcessedFilesCommand.php:34ReplyI have removed the other both optionsCleanProcessedFilesCommand.php:75ReplyI have removed this part.CleanProcessedFilesCommand.php:97ReplyDoneCleanProcessedFilesCommand.php:115ReplyI have added a join in DB statementCleanProcessedFilesCommand.php:119ReplyDoneCleanProcessedFilesCommand.php:153ReplyDoneCleanProcessedFilesCommand.php:157ReplyI have reworked this part and created a generator method to reduce memory consuption. -
TYPO3com voted on patch set 24 V -1 09/30/20, 22:19 Completed build in 20m 46s on Wed, 30 Sep, 10:19 PM Test Summary: 6 of 74184 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6108
-
Stefan Froemken uploaded patch set 25 10/01/20, 07:52
-
TYPO3com voted on patch set 25 V -1 10/01/20, 08:05 Completed build in 13m 15s on Thu, 1 Oct, 08:05 AM Test Summary: 3 of 74184 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6114
-
Stefan Froemken uploaded patch set 26 10/07/20, 19:03
-
TYPO3com voted on patch set 26 V -1 10/07/20, 19:20 Completed build in 16m 31s on Wed, 7 Oct, 07:20 PM Test Summary: 73797 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6338
-
Stefan Froemken uploaded patch set 27 10/07/20, 19:29
-
TYPO3com voted on patch set 27 V +1 10/07/20, 19:40 Completed build in 11m 40s on Wed, 7 Oct, 07:40 PM Test Summary: 73797 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6339
-
Josef Glatz wrote 10/08/20, 04:16
Removed reviewer Josef Glatz. -
Oli Bartsch commented on patch set 27 CR -1 10/08/20, 09:16
Feature-58721-CLI-Command-to-clean-unused-processed-images.rst:22database-only and files-only are no longer available.CleanProcessedFilesCommand.php:34Please declare visibilityCleanProcessedFilesCommand.php:37protected function configure(): voidCleanProcessedFilesCommand.php:62protected Personally I would also suggest, the execute method should only initialize the args / options and then call a service to handle them, like it's done with other recently added commands e.g. CheckIntegrityCommand or CleanupRedirectsCommand. -
Stefan Froemken uploaded patch set 28 10/08/20, 18:56
-
TYPO3com voted on patch set 28 V -1 10/08/20, 18:57 Completed build in 22s on Thu, 8 Oct, 06:57 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6384
-
Stefan Froemken commented on patch set 28 10/08/20, 18:59
Feature-58721-CLI-Command-to-clean-unused-processed-images.rst:22ReplyI have removed these linesCleanProcessedFilesCommand.php:34ReplyDoneCleanProcessedFilesCommand.php:37ReplyDoneCleanProcessedFilesCommand.php:62ReplyI have moved the logic into a new service -
Stefan Froemken uploaded patch set 29 10/08/20, 19:03
-
TYPO3com voted on patch set 29 V -1 10/08/20, 19:04 Completed build in 23s on Thu, 8 Oct, 07:04 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6386
-
Stefan Froemken uploaded patch set 30 10/08/20, 19:05
-
TYPO3com voted on patch set 30 V +1 10/08/20, 19:18 Completed build in 12m 35s on Thu, 8 Oct, 07:18 PM Test Summary: 73773 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6387
-
Oli Bartsch commented on patch set 30 V -1 10/09/20, 09:57
Idea of the service was to be able to reuse them without any command specific functionality added. Will provide a new patchset. -
Oli Bartsch uploaded patch set 31 10/09/20, 10:34
-
Oli Bartsch uploaded patch set 32 10/09/20, 10:39
-
TYPO3com voted on patch set 31 V +1 10/09/20, 10:47 Completed build in 12m 52s on Fri, 9 Oct, 10:47 AM Test Summary: 73773 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6396
-
TYPO3com voted on patch set 32 V +1 10/09/20, 10:52 Completed build in 13m 03s on Fri, 9 Oct, 10:52 AM Test Summary: 73773 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6397
-
Oli Bartsch uploaded patch set 33 10/09/20, 11:26
-
Oli Bartsch uploaded patch set 34 10/09/20, 11:30
-
TYPO3com voted on patch set 33 V +1 10/09/20, 11:38 Completed build in 12m 25s on Fri, 9 Oct, 11:38 AM Test Summary: 73773 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6398
-
TYPO3com voted on patch set 34 V +1 10/09/20, 11:44 Completed build in 13m 57s on Fri, 9 Oct, 11:44 AM Test Summary: 73773 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6399
-
Stefan Froemken commented on patch set 34 V +1 CR +1 10/09/20, 20:13
Thank you for your update -
Stefan Froemken commented on patch set 34 10/09/20, 20:15
Just for information: The files will be deleted, but not the file structure 0/a, 2/d and so on, which is totally Ok for me. -
Guido Schmechel commented on patch set 34 10/14/20, 15:06
In CleanProcessedFilesService.php line 175: "Directory name must not be empty." via $folder->getPublicUrl() Context: Created a storage for patch testing which is outside the docroot and the status at is_public is false. Known from https://github.com/Leuchtfeuer/typo3-secure-downloads/issues/91 var_dump($folder->getStorage()->getConfiguration()['basePath']); var_dump($folder->getStorage()->getProcessingFolder()->getIdentifier()); Maybe a combination from these two parameters could be a solution for the "is_public=false" case? But basically the code looks solid and without any special cases it runs fine. -
Guido Schmechel commented on patch set 34 10/14/20, 15:11
I see: $processedFilePath = $folder->getStorage()->getConfiguration()['basePath'] . $folder->getIdentifier(); should be enough. The processingFolder is already given ;-) -
Jonas Eberle commented on patch set 34 12/15/20, 20:58
Would it be possible to delete _processed files that are out of date, too? -
Benni Mack uploaded patch set 35 01/06/21, 23:06
-
TYPO3com voted on patch set 35 V -1 01/06/21, 23:17 Completed build in 11m 08s on Wed, 6 Jan, 11:17 PM Test Summary: 31850 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-8712
-
TYPO3com voted on patch set 35 V +1 01/08/21, 13:26 Completed build in 08m 54s on Fri, 8 Jan, 01:26 PM Test Summary: 32578 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-8763
-
Guido Schmechel voted on patch set 35 CR -1 01/09/21, 12:54
-
Tymoteusz Motylewski commented on patch set 35 01/09/21, 14:14
it would be nice to have a limit of files to process in one run. im also weonsering whether dry run shouldnt be the default behaviour and removig the files under some option -
Benni Mack wrote 11/29/21, 10:03
Change destination moved from master to main -
Benni Mack updated the commit message in patch set 36 11/29/21, 13:14
-
Oli Bartsch uploaded patch set 37 11/29/21, 13:22
-
core-ci voted on patch set 37 V -1 11/29/21, 13:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20015
-
Benni Mack rebased onto patch set 38 12/10/21, 10:46
-
core-ci voted on patch set 38 V -1 12/10/21, 10:50 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20924
-
Benni Mack uploaded patch set 39 12/10/21, 10:52
-
core-ci voted on patch set 39 V -1 12/10/21, 11:08 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20925
-
Benni Mack wrote 12/10/21, 11:47
Removed reviewer Benni Mack. -
Stefan Froemken uploaded patch set 40 12/23/21, 21:08
-
core-ci voted on patch set 40 V -1 12/23/21, 21:28 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21442
-
Stefan Froemken uploaded patch set 41 12/23/21, 22:13
-
Stefan Froemken commented on patch set 41 12/23/21, 22:16
> Maybe a combination from these two parameters could be a solution for the "is_public=false" case? I have implemented the code of LocalDriver::calculateBasePath into this patch. This is more safe. -
core-ci voted on patch set 41 V -1 12/23/21, 22:16 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21443
-
Stefan Froemken commented on patch set 41 12/23/21, 22:17
> $processedFilePath = $folder->getStorage()->getConfiguration()['basePath'] . $folder->getIdentifier(); should be enough. The processingFolder is already given ;-) What if basePath is not given. I have added a new method to get the basePath the safe way ;-) -
Stefan Froemken commented on patch set 41 12/23/21, 22:18
Reply
I have added an option to limit the files/records -
Stefan Froemken commented on patch set 41 12/23/21, 22:18
Feature-58721-CLI-Command-to-clean-unused-processed-images.rst:22ReplyAckCleanProcessedFilesCommand.php:62ReplyAck -
Stefan Froemken uploaded patch set 42 12/23/21, 22:20
-
Stefan Froemken uploaded patch set 43 12/23/21, 22:21
-
core-ci voted on patch set 42 V -1 12/23/21, 22:23 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21444
-
core-ci voted on patch set 43 V -1 12/23/21, 22:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21445
-
Stefan Froemken uploaded patch set 44 12/23/21, 22:27
-
core-ci voted on patch set 44 V -1 12/23/21, 22:41 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21446
-
Stefan Froemken uploaded patch set 45 12/24/21, 00:41
-
core-ci voted on patch set 45 V +1 12/24/21, 00:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21450
-
Stefan Bürk commented on patch set 45 01/07/22, 12:59
CleanProcessedFilesService.php:81Would suggest to change this to `$localStorages`CleanProcessedFilesService.php:103Would suggest to move this before the loops and create a prepared statement there, and reuse that pepared statement here with just the `identifier` as bind value. => 'QueryBuilder->prepare()'. See: 72610: [BUGFIX] Forward-compatible prepared statement support | https://review.typo3.org/c/Packages/TYPO3.CMS/+/72610 There are tests, maybe that is enough as this not documented yet (it's on the todo list). If needed support on this, let me know.CleanProcessedFilesService.php:181Would suggest to move this before the while loop and create a prepared statement there, and reuse that pepared statement here with just the uid as bind value. 'QueryBuilder->prepare()'. See: 72610: [BUGFIX] Forward-compatible prepared statement support | https://review.typo3.org/c/Packages/TYPO3.CMS/+/72610 There are tests, maybe that is enough as this not documented yet (it's on the todo list). If needed support on this, let me know. -
Stefan Bürk uploaded patch set 46 01/14/22, 22:59
-
Stefan Bürk commented on patch set 46 01/14/22, 23:04
Changed stuff: * used prepared statements for recurring queries only changing where values * rename variable to matching holding multiple storages and not only one * avoid using internal/deprecated execute() method and use proper executeQuery() or executeStatement() * avoid reusing QueryBuilder for multiple queriesm thus following recommendation for QueryBuilder usageCleanProcessedFilesService.php:81ReplyDoneCleanProcessedFilesService.php:103ReplyDoneCleanProcessedFilesService.php:181ReplyDone -
core-ci voted on patch set 46 V -1 01/14/22, 23:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22101
-
core-ci voted on patch set 46 V +1 01/14/22, 23:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22101
-
Stefan Bürk rebased onto patch set 47 01/25/22, 08:43
-
core-ci voted on patch set 47 V +1 01/25/22, 08:55 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22504
-
Georg Ringer commented on patch set 47 CR -1 02/09/22, 12:34
Feature-45039-CommandToCleanUpProcessedFiles.rst:13if this comment "Please note that the command currently only works for local drivers." is true. I would really change the title of the rst file and maybe even also the name of the command to "cleanup:localProcessedFiles" to communicate it. this important restriction must not be hidden in the last line of the rst file ;)Feature-45039-CommandToCleanUpProcessedFiles.rst:19shouldn't this be "with"Feature-45039-CommandToCleanUpProcessedFiles.rst:23I would rephrase because the name of the directory can be configured to something like: Also files in the configured temporary directory (typically `_processed_`) will be deleted if there are no references on them. -
Simon Schaufelberger commented on patch set 47 04/26/22, 20:32
@Eike, can you please take care about the open issues? Would be great to merge this soon. -
Stefan Froemken uploaded patch set 48
CR -1V +101/21/23, 10:23 -
core-ci voted on patch set 48 V -1 01/21/23, 10:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/36407
-
Stefan Froemken uploaded patch set 49
V -101/21/23, 10:38 -
Stefan Froemken uploaded patch set 50 01/21/23, 10:44
-
core-ci voted on patch set 49 V -1 01/21/23, 10:50 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/36408
-
core-ci voted on patch set 50 V -1 01/21/23, 10:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/36409
-
Stefan Froemken uploaded patch set 51
V -101/21/23, 11:04 -
core-ci voted on patch set 51 V +1 01/21/23, 11:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/36410
-
Stefan Froemken commented on patch set 51 01/21/23, 11:19
Feature-45039-CommandToCleanUpProcessedFiles.rst:13ReplyDoneFeature-45039-CommandToCleanUpProcessedFiles.rst:19ReplyDoneFeature-45039-CommandToCleanUpProcessedFiles.rst:23ReplyDone -
Stefan Bürk commented on patch set 51 01/21/23, 13:12
CleanUpLocalProcessedFilesService.php:134We do not use `\PDO::*` anymore (deprecated by doctrine/dbal) .. should be `Connection::PARAM_INT` -
Stefan Froemken uploaded patch set 52
V +101/21/23, 16:01 -
Stefan Froemken commented on patch set 52 01/21/23, 16:02
CleanUpLocalProcessedFilesService.php:134ReplyDone -
core-ci voted on patch set 52 V +1 01/21/23, 16:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/36411
-
Chris Müller uploaded patch set 53
V +102/08/23, 17:52 -
core-ci voted on patch set 53 V +1 02/08/23, 18:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/37401
-
Jochen Roth uploaded patch set 54
V +103/14/23, 12:02 -
Jochen Roth uploaded patch set 55 03/14/23, 12:08
-
core-ci voted on patch set 54 V -1 03/14/23, 12:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38915
-
core-ci voted on patch set 55 V -1 03/14/23, 12:20 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38916
-
Jochen Roth uploaded patch set 56
V -103/14/23, 12:20 -
core-ci voted on patch set 56 V +1 03/14/23, 12:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38919
-
Stefan Froemken commented on patch set 56 03/17/23, 12:44
CleanUpLocalProcessedFilesCommand.php:89In both cases I prefer $files === [] and $records === [] instead of countCleanUpLocalProcessedFilesCommand.php:111Are you sure this will work? This option is configured with VALUE_NONE and the default value is null. IMO this can never be true. -
Jochen Roth uploaded patch set 57
V +103/17/23, 13:04 -
Jochen Roth commented on patch set 56 03/17/23, 13:06
CleanUpLocalProcessedFilesCommand.php:89ReplyDoneCleanUpLocalProcessedFilesCommand.php:111Reply`--force` it returns true and false if not set. -
core-ci voted on patch set 57 V +1 03/17/23, 13:10 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39124
-
Jochen Roth uploaded patch set 58
V +103/17/23, 13:30 -
core-ci voted on patch set 58 V +1 03/17/23, 13:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39126
-
Guido Schmechel commented on patch set 58 03/19/23, 15:08
Basically, everything went in the cli as it should. However, I still have 1-2 questions and therefore no voting for the time being :-)CleanUpLocalProcessedFilesCommand.php:60I get the following message the first time I run the scheduler command: "Execution of task "1" failed with the following message: Cannot retrieve helper "question" because there is no HelperSet defined. Did you forget to add your command to the application or to set the application on the command using the setApplication() method? You can also set the HelperSet directly using the setHelperSet() method." The trigger is that "force" is not set. Presumably we can't set this per default for the scheduler, can we? Then I would at least add a note somewhere about what is necessary for the scheduler. Either here or in the RST.CleanUpLocalProcessedFilesService.php:41I'm just asking to be sure: Do we have to respond to any of the is_* settings in the file storage? -
Jochen Roth uploaded patch set 59
V +103/20/23, 10:29 -
Jochen Roth commented on patch set 58 03/20/23, 10:32
Thanks for your review!CleanUpLocalProcessedFilesCommand.php:60ReplyChanged it so if --no-interaction (which is set when run through the scheduler), --force is set as well so it's working now.CleanUpLocalProcessedFilesService.php:41ReplyNo, i don't think we have to respond to any of the is_* settings as we operate on the processed files.
-
core-ci voted on patch set 59 V -1 03/20/23, 10:39 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39228
-
Jochen Roth uploaded patch set 60
V -103/20/23, 11:10 -
core-ci voted on patch set 60 V +1 03/20/23, 11:20 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39231
-
Guido Schmechel commented on patch set 60 V +1 CR +1 03/20/23, 20:28
Thanks! :-) -
Benni Mack commented on patch set 60 03/21/23, 09:59
CleanUpLocalProcessedFilesService.php:198vfs is gone, we can remove this -
Jochen Roth uploaded patch set 61
CR +1V +103/21/23, 10:41 -
core-ci voted on patch set 61 V +1 03/21/23, 10:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39314
-
Jochen Roth commented on patch set 61 03/21/23, 10:49
CleanUpLocalProcessedFilesService.php:198ReplyDone -
Guido Schmechel voted on patch set 61 V +1 CR +1 03/21/23, 16:21
-
Stefan Bürk commented on patch set 61 CR -1 03/22/23, 12:18
on it ...Feature-45039-CommandToCleanUpLocalProcessedFiles.rst:31Should be two spaces between ".." and "code-block" to be aligned with 4-space intendation of the code blockCommandCest.php:60nitpick - looks nice to have it sorted up and be in alphabetical order ...CleanUpLocalProcessedFilesCommand.php:70superflous. does not contain more value than native type declation. can be removed.CleanUpLocalProcessedFilesCommand.php:163superflous. does not contain more value than native type declation. can be removed.CleanUpLocalProcessedFilesService.php:124@param is superflous. does not contain more value than native type declation. can be removed.CleanUpLocalProcessedFilesService.php:156superflous. does not contain more value than native type declation. can be removed.CleanUpLocalProcessedFilesTest.php:60Not happy with this (strict order) requirement. Tests should be work in any order. Granted, TF does not support easy things for now to ensure this per configuration. But we should not relay on this ordering. Will look if I can come up with a better fitting solution for this ....CleanUpLocalProcessedFilesTest.php:75* Use '//' comments for non-docblock comments.... * @todo (lower casing) * The link to the PR is not needed here and does not provide real value here. Should be removed.sys_file_processedfile.csv:2we should use proper quoting in (new) csv files for * table names * column names * string values (e.g. filenames, text, title etc) TF importer can deal with it, but it would eventually avoid issues in the future for changes.oneDeleted.csv:1we should use proper quoting in (new) csv files for * table names * column names * string values (e.g. filenames, text, title etc) TF importer can deal with it, but it would eventually avoid issues in the future for changes. -
Stefan Bürk uploaded patch set 62
CR +1V +103/22/23, 13:12 -
Stefan Bürk commented on patch set 62 03/22/23, 13:12
Feature-45039-CommandToCleanUpLocalProcessedFiles.rst:31ReplyDoneCommandCest.php:60ReplyDoneCleanUpLocalProcessedFilesCommand.php:70ReplyDoneCleanUpLocalProcessedFilesCommand.php:163ReplyDoneCleanUpLocalProcessedFilesService.php:124ReplyDoneCleanUpLocalProcessedFilesService.php:156ReplyDoneCleanUpLocalProcessedFilesService.php:158we can rename this method to getAbsoluteBasePath() which makes the comment superflous too.CleanUpLocalProcessedFilesService.php:185We should rename this to to reflect the (partial) prepared state, e.g. `getQueryBuilderWithoutRestrictions()` )CleanUpLocalProcessedFilesTest.php:60ReplyMoved it intentionally to the end. Redoing file providing to instance on tear down now for only this testcase. This ensures files are there for each test, not only the whole test-case.CleanUpLocalProcessedFilesTest.php:75ReplyThat's not a @todo at all, or ? Because it already tests if files are still exists - so the todo is done.sys_file_processedfile.csv:2ReplyDoneoneDeleted.csv:1ReplyDone -
core-ci voted on patch set 62 V +1 03/22/23, 13:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39383
-
Stefan Bürk commented on patch set 62 CR -1 03/22/23, 13:51
On my local system, this always deletes processed files, even if references are still there and valid. This breaks then pages loaded from cache. E.g. the query to check if storage files has process file informations uses the storage name for fetching it from `sys_file_processedfile` as identifier, which is wrong. It's saved without the storage folder in it there correct identifier: '/_processed_/xxxxx' lookup identifier: '/fileadmin/_processed_/xxx' Therefore, it always says that the file need's to be deleted. * issue must be solved * we need additional tests to cover these cases -
Stefan Bürk commented on patch set 62 03/22/23, 19:22
on it ..CleanUpLocalProcessedFilesTest.php:29Current tests only deals with "relative" local storage - without a correct file storage setup. Missing/should be done: * add storage creation during `setUp` for fileadmin (relative) local storage * add storage creaton during `setUp` for an additional "absolute" local storage * extend tests to cover additional storagesCleanUpLocalProcessedFilesTest.php:111Should also check against `deleted files` count.CleanUpLocalProcessedFilesTest.php:124Should check against not containing "Deleted" only to ensure no deletion have been done. With that exact string test would ge happy if it would say "Deleted 2 processed records"sys_file_processedfile.csv:3test setup has no file storage 1 (fileadmin) - should be created -
Stefan Bürk uploaded patch set 63
CR -1V +103/22/23, 19:39 -
Stefan Bürk commented on patch set 63 03/22/23, 19:39
Reply
DoneCleanUpLocalProcessedFilesTest.php:29ReplyDoneCleanUpLocalProcessedFilesTest.php:111ReplyDoneCleanUpLocalProcessedFilesTest.php:124ReplyDonesys_file_processedfile.csv:3ReplyDone -
Stefan Bürk updated the commit message in patch set 64 03/22/23, 19:54
-
core-ci voted on patch set 64 V +1 03/22/23, 20:03 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39415
-
Stefan Bürk commented on patch set 64 V +1 CR +1 03/22/23, 20:08
I would say, this change is now in a reasonable shape. Did some minor adjustments, fixing a oversight and added little more testing to it - beside ensuring a better suited test setup. -
Benni Mack uploaded patch set 65
CR +1V +103/22/23, 20:27 -
Benni Mack voted on patch set 65 V +1 CR +1 03/22/23, 20:27
-
core-ci voted on patch set 65 V +1 03/22/23, 20:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39417
-
Stefan Bürk voted on patch set 65 V +2 CR +2 03/22/23, 20:38
-
Stefan Bürk merged patch set 66 03/22/23, 20:38
CleanProcessedFiles.php:7