Review main Merged

50371[BUGFIX] Don't set resource storages permanently offline automatically

Marc Willmann CR +2 V +2 Normal +48 −3 updated 12/01/16, 12:34

Commit message

When the storage drive coudn't be initialized correctly the storage is
only put offline for the current request. The error thrown by the driver
is logged to database table sys_log as well.

Review

  • Code review CR +2
    • Anja Leichsenring +2
    • Oliver Hader +1
  • Verified V +2
    • Anja Leichsenring +2
    • Oliver Hader +1
    • TYPO3com +1 CI

Issues 1

Files 2

Review log 4758

Show 2533 earlier messages
  1. Marc Willmann uploaded patch set 1 10/25/16, 15:22

  2. TYPO3com voted on patch set 1 V +1 10/25/16, 15:27 Completed build in 04m 43s on Tue, 25 Oct, 05:27 PM Test Summary: 19569 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4170

  3. Stefan Bürk commented on patch set 2 10/25/16, 17:21

    Published edit on patch set 1.
  4. Stefan Bürk voted on patch set 2 CR +1 V +1 10/25/16, 17:21

  5. TYPO3com voted on patch set 2 V +1 10/25/16, 17:27 Completed build in 06m 05s on Tue, 25 Oct, 07:27 PM Test Summary: 19569 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4182

  6. Marc Willmann uploaded patch set 3 10/26/16, 17:59

  7. TYPO3com voted on patch set 3 V +1 10/26/16, 18:02 Completed build in 03m 52s on Wed, 26 Oct, 08:02 PM Test Summary: 19569 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4268

  8. Frans Saris commented on patch set 3 10/26/16, 18:49

    Can you update the commit message so it reflects current change
  9. Marc Willmann updated the commit message in patch set 4 10/27/16, 07:26

  10. Marc Willmann commented on patch set 4 10/27/16, 07:28

    thanks for your hint, Frans. Commit message is updated. Are you confirm with the patch itself?
  11. TYPO3com voted on patch set 4 V +1 10/27/16, 07:30 Completed build in 04m 06s on Thu, 27 Oct, 09:30 AM Test Summary: 19569 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4285

  12. Frans Saris commented on patch set 4 10/27/16, 07:36

    ResourceStorage.php:184

    Please add some logging so the admin has some clue what happened. if ($this->getBackendUser()) { $this->getBackendUser()->writelog(..., $e->getMessage()); }
  13. Marc Willmann uploaded patch set 5 10/27/16, 10:05

  14. TYPO3com voted on patch set 5 V +1 10/27/16, 10:15 Completed build in 10m 05s on Thu, 27 Oct, 12:15 PM Test Summary: 19569 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4307

  15. Christian Kuhn updated the commit message in patch set 6 10/27/16, 11:06

  16. TYPO3com voted on patch set 6 V +1 10/27/16, 11:11 Completed build in 05m 20s on Thu, 27 Oct, 01:11 PM Test Summary: 19569 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4314

  17. Frans Saris commented on patch set 6 10/27/16, 11:31

    Not sure if it makes sense to at the logging timeout. Now the admin has to wait 5 minutes after changing the storage config to see a new config error message.
  18. Marc Willmann commented on patch set 6 10/27/16, 11:39

    without logging timeout the syslog is FULL of messages in a very short time. Maybe this will increase the problem. After changing the configuration the admin may see a result immediately (resource storage is working or not) - but not in the log, this is correct.
  19. Philipp Gampe commented on patch set 6 10/31/16, 14:26

    While the idea and the problem of the patch are correct, I do not think the solution is fully satisfactory yet. It is correct that a storage might come back online (e.g. after a short network loss) any time and should start working automatically again. However it is also true, that a temporary offline storage should not exhaust the systems resources (30 sec hang, open processes, ports, etc). That means there must be a way to tell the system that a storage is temporary unavailable without having to check the real connection. Therefore I suggest to introduce two new fields: * temporary_offline (bool) * reconnect_wait_time (int) default wait time for a reconnect in seconds (default: 60) This would move the wait time logic from the logging to the actual buisiness logic of the remote storage connection. Optionally provide another field with the option to choose what to do in case of connection failure: * perma offline * perma offline on config error * temp offline * temp offline on config error * always online Default: temp offline In any way, this would be a Feature and therefore master only.
  20. Philipp Gampe voted on patch set 6 CR -1 10/31/16, 14:26

  21. Marc Willmann commented on patch set 6 10/31/16, 18:09

    The idea to take the resource storage temporary offline isn't new; in fact the described behaviour is near the solution provided in patchset 2. I don't have a problem with this; but I think the problem is too important to solve it "somewhere in the future". A (future) design should be IMHO that the given driver for a resource storage decides, if it makes sense to put a resource temporary or permanent offline and for what time period. For a local storage the answer will be likely "no", for a S3 or WebDav-resource may be "yes, for ... secs". As said before I'm voting to not mix up the BUGFIX for #75184 (for which this is a better solution than before) and a FEATURE for another ticket (which will bring a LOT of discussion). Yes, the resource storage implementation should not be forgotten after this patch, but I would like to see working code in this important area of TYPO3.
  22. Frans Saris commented on patch set 6 CR -1 11/15/16, 12:11

    I will push a new patch dropping the interval for the logging part. As this doesn't make sense imo. If it's broken it's broken. Cleaning the log after it is fixed isn't that hard. And for other places we also don't add such mechanisms. The idea of Philipp sounds good but is a new feature and should block this issue as it will not be a solution for 7.6
  23. Frans Saris uploaded patch set 7 11/15/16, 12:34

  24. Frans Saris updated the commit message in patch set 8 11/15/16, 12:34

  25. Frans Saris voted on patch set 8 CR +1 V +1 11/15/16, 12:34

  26. TYPO3com voted on patch set 7 V +1 11/15/16, 12:42 Completed build in 08m 29s on Tue, 15 Nov, 01:42 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4930

  27. TYPO3com voted on patch set 8 V +1 11/15/16, 12:49 Completed build in 14m 57s on Tue, 15 Nov, 01:49 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4931

  28. Markus Klein voted on patch set 8 CR +1 11/15/16, 16:37

  29. Philipp Gampe voted on patch set 8 CR +1 11/16/16, 15:55

  30. Marc Willmann voted on patch set 8 CR +1 V +1 11/16/16, 16:04

  31. Sascha Egerer commented on patch set 8 11/16/16, 19:26

    ResourceStorage.php:189

    "initialising" (british english) should be "initializing"

    ResourceStorage.php:405

    This method is not called anymore but the core but is public API. should we deprecate it?
  32. Frans Saris commented on patch set 8 11/16/16, 19:29

    ResourceStorage.php:405 Reply

    Deprecation will be done in a separate patch once the final API is clear for disabling storage by driver.
  33. Marc Willmann commented on patch set 8 11/16/16, 19:29

    ResourceStorage.php:405 Reply

    Both setTemporaryOffline and setPermantlyOffline is not called anymore and should be deprecated. But I don't think it's a good idea to do so in THIS patch, but better in the follow-up redesign of this code.
  1. Sascha Egerer commented on patch set 9 11/16/16, 19:31

    Published edit on patch set 8.
  2. Sascha Egerer voted on patch set 9 CR +1 11/16/16, 19:31

  3. Marc Willmann voted on patch set 9 CR +1 11/16/16, 19:32

  4. TYPO3com voted on patch set 9 V +1 11/16/16, 19:35 Completed build in 04m 05s on Wed, 16 Nov, 08:35 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4939

  5. Sascha Egerer commented on patch set 9 CR -1 11/16/16, 19:41

    ResourceStorage.php:189

    The exception looks like this Failed initializing storage [3], error: " does not exist or is no directory. So looks like the exception message is wrong but this is a different issue. But I'm also not happy with the logging. Why do we only log if we have a backend user? We should use LogManager instead like i did in the reverted patch (https://review.typo3.org/#/c/49798/7/typo3/sysext/core/Classes/Resource/ResourceStorage.php). What do you think?
  6. Frans Saris commented on patch set 9 11/16/16, 19:50

    ResourceStorage.php:189 Reply

    The problem with the LogManager I see is that it isn't visible in the sys_log available in the BE. Undecided if it makes sense to log it when not in BE context
  7. Sascha Egerer commented on patch set 9 11/16/16, 19:51

    Should we log to both? The problem is that you'll never know when a storage was offline. That could be very hard to debug.
  8. Frans Saris commented on patch set 9 11/16/16, 20:08

    let's go for both
  9. Frans Saris commented on patch set 9 CR -1 11/16/16, 20:11

    on it
  10. Frans Saris uploaded patch set 10 11/16/16, 20:35

  11. Marc Willmann voted on patch set 10 CR +1 11/16/16, 20:36

  12. Frans Saris voted on patch set 10 CR +1 V +1 11/16/16, 20:36

  13. TYPO3com voted on patch set 10 V +1 11/16/16, 20:39 Completed build in 04m 14s on Wed, 16 Nov, 09:39 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4940

  14. Jörg Bösche voted on patch set 10 CR +1 11/16/16, 20:46

  15. Stefan Neufeind commented on patch set 10 11/16/16, 22:14

    Do we need to log twice? Or can we skip double-logging for the backendUser-case?

    ResourceStorage.php:192

    what is log-type 5? I only found 3, 4 and 255 mentioned in the core as well the core "TYPO3 core API"
  16. Frans Saris commented on patch set 10 11/17/16, 10:36

    IMO we can log it twice. Are 2 different cases. I want to have the sys_log logging here because that is easy visible for the "normal" admin in the BE log section.

    ResourceStorage.php:192 Reply

    log-type 5 is error handler
  17. Gleb Levitin voted on patch set 10 CR +1 12/01/16, 09:44

  18. Oliver Hader commented on patch set 10 12/01/16, 10:02

    Not sure about the double logging...
  19. Oliver Hader uploaded patch set 11 12/01/16, 11:07

  20. Oliver Hader commented on patch set 11 CR +1 V +1 12/01/16, 11:09

    Double logging can be configured now (enabled by default). The data-attributes have been json-encoded by the logging-framework, thus proper serialization into a string now happens in ResourceStorage already.
  21. TYPO3com voted on patch set 11 V +1 12/01/16, 11:14 Completed build in 06m 54s on Thu, 1 Dec, 12:14 PM Test Summary: 54077 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5230

  22. Anja Leichsenring voted on patch set 11 CR +2 V +2 12/01/16, 11:20

  23. Gerrit merged patch set 11 12/01/16, 11:20

  24. Anja Leichsenring commented on patch set 12 12/01/16, 11:21

    Cherry Picked This patchset was cherry picked to branch TYPO3_7-6 as commit 7c2a834b827c98675f67460556b91ec5d024fb53
  25. Frans Saris commented on patch set 12 12/01/16, 12:34

    ResourceStorage.php:3032

    This isn't uses so should be removed