36446[BUGFIX] Ensure temporary copies of files will be deleted by the driver
Commit message
Review
-
Code review CR 0
-
Verified V 0
Issues 2
Files 1
Review log 5253
Show 28 earlier messages
-
Thomas Deuling uploaded patch set 1 01/30/15, 16:14
-
Frans Saris commented on patch set 1 01/30/15, 20:59
-
Sascha Egerer commented on patch set 1 01/30/15, 21:27
LocalPreviewHelper.php:98ReplyI think that as not required because the filename is so unique that it is nearly impossible to have a file with this name in your storage i think. But to be sure we can also search for the typo3temp folder name -
Frans Saris commented on patch set 1 01/31/15, 09:03
LocalPreviewHelper.php:98ReplyDiscussed this with Thomas yesterday and I agree here. But would be good to add this in the comment so the next one that has a look here knows what happens here -
Thomas Deuling uploaded patch set 2 01/31/15, 13:32
-
Thomas Deuling uploaded patch set 3 01/31/15, 13:33
-
Frans Saris commented on patch set 3 CR +1 V +1 01/31/15, 17:34
You could set $originalFileName = $sourceFile->getForLocalProcessing(FALSE); to TRUE to easily verify this. -
Michael Oehlhof commented on patch set 3 02/15/15, 00:02
Tried to reproduce/verify with the dropbox driver. There are already fal-tempfile-* files created in typo3temp, which were not deleted. -
Michael Oehlhof commented on patch set 3 V +1 CR +1 02/15/15, 00:12
OK, I have debugged the code and the file is deleted. The remaining files must come from other places in the code. -
Michael Oehlhof uploaded patch set 4 02/15/15, 20:46
-
Michael Oehlhof commented on patch set 3 02/15/15, 20:48
As mentioned in bug 62098, all temporary created files are deleted in the destructor now. -
Michael Oehlhof commented on patch set 4 CR +1 V +1 02/15/15, 20:48
As mentioned in bug 62098, all temporary created files are deleted in the destructor now. -
Andy Grunwald commented on patch set 4 V -1 02/16/15, 20:52
Github tests: failure Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/22 -
Michael Oehlhof uploaded patch set 5 02/16/15, 22:43
-
Michael Oehlhof commented on patch set 5 CR +1 V +1 02/16/15, 22:44
Adjust UnitTest -
Andy Grunwald commented on patch set 5 V +1 02/17/15, 07:17
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/32 Service: continuous-integration/travis-ci Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51008677 -
Frans Saris commented on patch set 5 CR -1 V -1 02/17/15, 09:18
This is to gready. If I want a write able file I do not expect it to be removed magicly. This should only hapoen for getForLocalProcessing(false) the true value wouldn't make sence otherwiseAbstractDriver.php:82Check if file still exists -
Michael Oehlhof uploaded patch set 6 02/17/15, 12:25
-
Michael Oehlhof commented on patch set 6 CR +1 V +1 02/17/15, 12:34
Moved the list of files to be deleted from the driver to the file object. -
Frans Saris commented on patch set 6 CR -1 02/17/15, 13:17
I'm not sure it there is currently a central place to handle thisAbstractFile.php:542this will remove original files as for local files the returned path of getFileFolLocalProcessing is the path to the original file -
Andy Grunwald commented on patch set 6 V -1 02/17/15, 13:47
Github tests: failure Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/35 -
Michael Oehlhof uploaded patch set 7 02/17/15, 22:06
-
Michael Oehlhof commented on patch set 7 02/17/15, 22:08
Next try, Frans. Moved the logic to the resource storage. -
Andy Grunwald commented on patch set 7 02/17/15, 22:35
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/42 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51144360 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51145132 -
Andy Grunwald commented on patch set 7 02/17/15, 22:50
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/43 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51144360 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51147891 -
Andy Grunwald commented on patch set 7 02/18/15, 12:46
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/55 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51144360 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51221732 -
Andy Grunwald commented on patch set 7 02/18/15, 13:27
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/61 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51144360 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51226364 -
Andy Grunwald commented on patch set 7 02/18/15, 13:53
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/64 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51144360 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51229531
-
Frans Saris commented on patch set 7 CR -1 02/18/15, 18:24
Sorry this also isn't sufficient.ResourceStorage.php:1229This failes for non public local storages. -
Michael Oehlhof uploaded patch set 8 02/19/15, 20:21
-
Frans Saris commented on patch set 8 02/27/15, 07:25
ResourceStorage.php:1229I'm not sure about the basename() compare here. Need the opinion from a remote storage user(s) if this is save. -
Andy Grunwald commented on patch set 8 03/13/15, 19:52
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/68 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51425614 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/54292336 -
Andy Grunwald commented on patch set 8 05/14/15, 21:48
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/74 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51425614 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/62616208 -
Andy Grunwald commented on patch set 8 05/14/15, 21:59
Github tests: success Pull request: https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests/pull/78 Service: continuous-integration/travis-ci/push Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/51425614 Service: continuous-integration/travis-ci/pr Description: The Travis CI build passed URL: https://travis-ci.org/typo3-ci/TYPO3.CMS-pre-merge-tests/builds/62616208 -
Stefan Neufeind commented on patch set 8 05/22/15, 14:53
Hmm, how can we continue here? I just trapped into an ever-filling typo3temp as well. In this case files *were* local - but in a separate storage for "uploads/tx_myext" (for compat reason from a 4.5-installation). I'm still unsure why it seems to be recognised as "non-local" (and thus the temp-copy happens). -
Stefan Neufeind commented on patch set 8 CR -1 05/22/15, 15:14
ResourceStorage.php:1229ReplyIt's been mentioned in another discussion lately that the identifier can be "anything" (a hash, ...) depending on the storage. So it's not always the filename. And even if it were, comparing by name is not really save anyway imho. The idea is to check if we got a temporary filePath back? Can we check whether the dirname($filePath) matches the temporary dir then (the other way round)? -
Frans Saris commented on patch set 8 05/29/15, 09:52
Currently I do not see a proper solution for this. -
Oliver Hader uploaded patch set 9 10/09/15, 06:50
-
Stefan Neufeind commented on patch set 9 10/12/15, 12:29
How do we continue here? -
Frank Nägler commented on patch set 9 12/11/15, 12:50
ResourceStorage.php:203the ResourceStorage should never contain any file operation. file_Exists and unlink must be used only in the driver. -
Arno Schoon commented on patch set 9 CR +1 V +1 12/22/15, 15:45
We applied patch set 9 on top of our 6.2.15 platform hosted on AWS. Due to a remote storage of files (S3) the typo3temp folder on all four web servers flooded overnight, this patch did fix the issue for (from what I understand) our specific scenario. From my point of view the check using basename() is sufficient if a driver implementation uses the `\TYPO3\CMS\Core\Resource\Driver\AbstractDriver::getTemporaryPathForFile` method so the name of the temporary file meets the `fal-tempfile-` pattern. -
Frank Nägler voted on patch set 9 CR -1 01/15/16, 18:31
-
Helmut Hummel commented on patch set 9 CR -1 04/09/16, 19:54
I agree with Frank here. This must be fixed differently -
Frans Saris commented on patch set 9 CR -1 04/10/16, 06:46
Temp files aren't part of the driver, so using direct file operations. But not direct unlink but generalUtility unlinkTemp Mean problem here is that only the local driver uses the tempfile when the file needs to be writeable, it isn't API to use temp files when the file isn't available locally for only reading purposes. So this is a responsibility of the driver. Or we need to change the FAL API for instance that the file is returned as stream to the storage and that if needed the temp file is created there. -
Frans Saris commented on patch set 9 04/20/16, 16:45
the LocalDriver should keep a record of all temp files that are created by LocalDriver::copyFileToTemporaryPath() and delete them when still present in LocalDriver::__destroy() This needs to be documented/advertised then so that extension/fal driver authors also take care of the temp files. This isn't something the core can/should handle -
Nicole Hummel commented on patch set 9 04/20/16, 22:25
not 7.6? and are you sure this is critical enough to be worth to backport it to 6.2?ResourceStorage.php:158[] -
Frans Saris uploaded patch set 10 04/21/16, 12:27
-
Helmut Hummel commented on patch set 10 CR -1 04/21/16, 12:31
We should avoid such __destruct code. Imho we should rather find a way to cleanup stuff in a bootstrap shutdownLocalDriver.php:89in case of an insecure unserialize vulnerability, this would enable an exploit to delete arbitrary files. -
TYPO3com voted on patch set 10 V -1 12/04/16, 11:23 Completed build in 30m 29s on Sun, 4 Dec, 12:23 PM Test Summary: 6 of 54322 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5476
-
Benni Mack commented on patch set 10 02/06/18, 21:08
I agree with Helmut. Any care to update the patch? -
Frans Saris commented on patch set 10 05/16/18, 17:15
@helmut can you point me to an example how I could use the bootstrap shutdown for this?LocalDriver.php:89Replybut GeneralUtility::unlink_tempfile() makes sure only files inside of the tempfolder are removed. -
Helmut Hummel commented on patch set 10 05/18/18, 13:19
> can you point me to an example how I could use the bootstrap shutdown for this? currently there is no bootstrap shutdown any more Looking at this again, I think the driver should not delete files that are created with getForLocalProcessing. The callers of this method are responsible for deleting the file if it is not needed any more. This was done in one place already: https://review.typo3.org/#/q/I5d34574c1efccd12a85eb263c4397bc73a9516f3 If there are still temp files created, it means that we still call this API, without cleaning up after that. Looks like e.g. LocalCropScaleMaskHelper still does this multiple times, when masking the image. There might be additional places. What I want to say: The proposed change here hides wrong usages of the API and additionally raises concerns regarding security (even if the impact is low, due to the check in unlink_tempfile). -
Frans Saris abandoned the change 05/18/18, 14:13
The (php)docs state that you should take care of the temp files yourself when using this part of the API. So I agree that this belongs to (I'm) proper use of the API. So for now I abandoned this change. If we find improper use in the core this should be fixed in a separate issue.
LocalPreviewHelper.php:98