Review main Abandoned

52415[TASK] Fallback to TransientMemory + VariableFrontend cache

Claus Due CR 0 V 0 Easy +19 −7 updated 05/13/17, 10:10

Commit message

Instead of throwing an exception when a cache is not
configured, CacheManager can fallback to a universal
cache (TransientMemory + VariableFrontend) which
works as substitute for any configured cache.

The case is logged to devLog with NOTICE severity
as a means to track problems with things like typos
in cache identifier names.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 2

Review log 4344

Show 1819 earlier messages
  1. Claus Due uploaded patch set 1 04/11/17, 14:06

  2. TYPO3com voted on patch set 1 V +1 04/11/17, 14:15 Completed build in 09m 13s on Tue, 11 Apr, 04:15 PM Test Summary: 56642 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-10473

  3. Thomas Hohn voted on patch set 1 CR +1 04/11/17, 15:16

  4. Kasper Ligaard commented on patch set 1 CR +1 05/03/17, 06:42

    I Like: By inspecting log files I can now get a hint of caches to look into configuring; nice :-)
  5. Mads Lønne Jensen commented on patch set 1 CR +1 05/04/17, 10:50

    Look good, but perhaps we should consider setting the error level to WARNING?
  6. Claus Due commented on patch set 1 05/04/17, 11:14

    Re: setting level to WARNING that's not necessary imho. The fallback is safe for every backend except for the File and SimpleFile ones which are less frequently used than the others. Were it up to me there'd be no log message at all and this would be a normally exploitable feature-of-fact, but that's just a personal preference ;)
  7. Elmar Hinz commented on patch set 1 05/11/17, 11:06

    It doesn't convince me to hide this exception from direct output. Do you have a use case where this is of advantage?
  8. Claus Due commented on patch set 1 05/11/17, 11:38

    > Do you have a use case where this is of advantage? Yes, but you're going to have to think a bit outside the box on this one ;) many of the classes in TYPO3 will use the runtime cache, Extbase will use the object- and reflection caches, and others. This means these classes throw exceptions as soon as you try to use them outside of TYPO3. Now, if we add this patch, those classes and others simply won't require a bootstrapped configuration. To name a few things that become possible: * ObjectManager can create class instances w/o TYPO3 env * Several TYPO3 classes such as ViewHelpers can operate w/o TYPO3 env (because ReflectionService and ObjectManager both work with the fallback) This all ties into how I want to attack the "sane configuration defaults" issue. It seems that the preference is to require complete bootstrapping and put the "sane defaults" in the TYPO3-only configuration. My pragma would be to allow classes to fall back automatically without any TYPO3 configuration present. One of the reasons I'm aiming for this is to allow loading these classes outside of TYPO3, to inspect them - which is particularly good to be able to do in the case of ViewHelpers (schema generation, validation, decoupling for third-party use, and so on). So although this looks "bad" from the current standpoint and pragma behind TYPO3's "sane defaults", I sincerely argue that we need to be smarter about this if we're ever going to completely decouple our monolith.
  9. Elmar Hinz commented on patch set 1 05/11/17, 13:26

    I fully follow your philosophy of decoupling the libraries. But does this patch reach the goal? It logs to devLog with NOTICE severity. This should not happen for an intended behaviour. Actually you want to be able to use parts of TYPO3 independently, parts that require a cache. Then you should explicitely provide a cache instead of doing mystery things in the depth. I think the right approach is to inject a cache on the most upper level of each library. Then you can do this, when using the libraries independently and TYPO3 can do it when using the libraries internally.
  10. Claus Due commented on patch set 1 05/11/17, 13:32

    > But does this patch reach the goal? It does for the cases described. The main hinge here is to allow ObjectManager to load classes and so perform DI. > It logs to devLog with NOTICE severity. This should not happen for an intended behaviour. I quote myself :) > Were it up to me there'd be no log message at all and this would be a normally exploitable feature-of-fact, but that's just a personal preference ;) > I think the right approach is to inject a cache on the most upper level of each library. Then you can do this, when using the libraries independently and TYPO3 can do it when using the libraries internally. I respectfully disagree. Requiring injection of a cache is just one more way that we prevent our classes from functioning (without significant customisation) outside our own product. Our caches in turn depend on configuration and some pull in the database layer although that isn't necessary. The TL;DR is - these caches are compatible with the normal DB-based ones (and redis, and memcached, anything except for PHP file caches). Fallbacks are 100% safe. Only difference: these fallback caches will not persist data.
  11. Elmar Hinz commented on patch set 1 05/11/17, 13:44

    > I respectfully disagree. Requiring injection of a cache is just one more way that we prevent our classes from functioning (without significant customisation) outside our own product. Our caches in turn depend on configuration and some pull in the database layer although that isn't necessary. Cache injection makes the external user independent from our product and our configurations. The only requirement is, that his cache provides the right API to speak to. That is easy to add with a few lines of code to any cache he choices to use.
  12. Claus Due commented on patch set 1 05/11/17, 13:47

    > That is easy to add with a few lines of code to any cache he choices to use. This is the core of the problem. Having to manually initialise things to be able to use our classes, plus forcing people to integrate with our API. If/when at some point we manage to decouple TYPO3 classes and for example switch to doctrine classes I'll happily switch positions and agree it's okay to require configuration. But until that time, forcing our custom API upon people just isn't an option to me when there's a way that does not do that.
  13. Elmar Hinz commented on patch set 1 05/11/17, 13:55

    If there is a standard API of caches, it would be clever to use this for TYPO3. That doesn't change the logic, that giving the user the option to inject his caches gives him freedom and independency. You can still create default caches, when the user doesn't provide his own. However, this should not happen in mystery depths, but right in the outer method, where the user would regularly inject his own cache.
  14. Christian Kuhn voted on patch set 1 CR -2 05/11/17, 13:57

  15. Claus Due commented on patch set 1 05/11/17, 13:57

    @Elmar you know that overall I agree with that position, but it builds on things that just do not exist at this time. We've got no universal cache implementation; any cache you define still has to be a TYPO3-specific one. We have no entrypoint to define our caches other than the huge global TYPO3 array. And we're still forcing people to define these if we don't allow the cachemanager to fall back. One final argument from me would be that if anything can be called a "cache handling entry point" in TYPO3 then it's the CacheManager - and this is exactly where I chose to implement the fallback.
  16. Christian Kuhn commented on patch set 1 05/11/17, 14:01

    I'm not convinced of that idea: You add magic in a deep API level to solve issues that belong to some application above. You even figured that out on your own that this solution is not clean and added a "notice level warning" in such a fallback case. We should avoid such use cases, and avoid such magic cache creation: There either is a cache configured and available, or there isn't. Do not create them on the fly with some arbitrary default settings, just to throw logs in this case.
  17. Claus Due commented on patch set 1 05/11/17, 14:06

    Guys, you're stuck thinking inside the TYPO3 box. > You add magic in a deep API level to solve issues that belong to some application above. The point is: we simply do not have to force the application above to deal with our esoteric configuration requirements. > You even figured that out on your own that this solution is not clean and added a "notice level warning" in such a fallback case. You are simply interpreting this incorrectly. The logging is there to at least partially respect the fact that there used to be an exception - I also clearly state that it is there to allow tracking problems with things like typos (calling getCache with a typo'ed name no longer causes exception but code still works). > We should avoid such use cases, and avoid such magic cache creation: There either is a cache configured and available, or there isn't. Do not create them on the fly with some arbitrary default settings, just to throw logs in this case. Then by all means let's remove this logging, I'd rather be without it, as you can see. But don't argue this way that you do please - the logical consequence is we'll never make our classes friendly to outside consumption because they'll keep demanding all sorts of custom initialisation. This is the wrong choice imho, on a much higher level than just dealing with facilitating our own little application. Please reconsider. This isn't as much magic as it is a way to not demand so much custom preparation just to do things like allow ObjectManager to create class instances.
  18. Christian Kuhn commented on patch set 1 05/11/17, 14:11

    -2 will be kept, this still does not convince me: the cache is bound to configuration, it must not work without. throwing an exception here is good habit: Usually, caches are configured, that is how the API works. If a developer has a typo in the cache name, it is very good to explode with an exception. this avoids nasty and in this case hard to find and track programming errors "my application uses caches and is still slow, no idea why". this has nothing to do with 'out the box'. the configuration can be injected from the outside into the - currently typo3 internal and not in any way exported or extracted - caching framework.
  19. Claus Due commented on patch set 1 05/11/17, 14:17

    Then let me ask the question this way: why does it require configuring not one, but at least two different caches just to be able to load a TYPO3 class using ObjectManager? I know the technical reason, but what's the logical reason we would give to someone from the outside trying to, for example, generate ViewHelper documentation without first installing an entire TYPO3 site or creating a custom bootstrapping? Why can't a class be instanced without all this, especially given that this merge request proves it can with just one simple fallback...?
  1. Claus Due commented on patch set 1 05/11/17, 14:19

    ...and *please* don't reply "if you want to generate XSDs you should know you need to install TYPO3" because that's exactly what I'm trying to convince you to let us avoid.
  2. Christian Kuhn commented on patch set 1 05/11/17, 14:48

    i don't get the point. what are you talking about here? fluid standalone has no dependency to typo3 core whatsoever, so VH instantiation can't be an issue here. the object manager is bound to extbase which is bound to typo3 core, and typo3 specific VH's are thus bound to typo3, which is then configured + bootstrapped properly.
  3. Claus Due commented on patch set 1 05/11/17, 14:58

    The specific chain of sad events that lead to this merge request: * https://github.com/TYPO3/Fluid.SchemaGenerator can generate schemas for Fluid just fine * add this lib as a --dev dependency and you expect to be able to generate schemas for TYPO3 ViewHelpers as well * but you can't So, why can't you do that? * Because first of all, TYPO3 ViewHelpers have to be instanciated with ObjectManager for the ReflectionService to be included. * This is necessary because the ReflectionService triggers when you do prepareArguments() to extract arguments (render method args analysed with reflection). * Loading any class with ObjectManager causes failures because of the cache. * ReflectionManager requires the second cache. This leaves us with a situation where to call an API method on a ViewHelper class, which is possible to call for any standard Fluid ViewHelper, has all of these requirements put onto the cache configuration. The result is that the XSD generator is unable to generate schemas for TYPO3 ViewHelpers. Now, this is where you might say "but doing this simply requires TYPO3 context" and that's where I would argue that this is wrong - pointing to the fact that a sane fallback truly does allow all of the above to work transparently. Please be aware that this is merely one among many similar use cases. This one use case is however immediately obvious with a nice example library that demonstrates the difference. Without this change: exceptions for *every* TYPO3 ViewHelper you attempt to schema. With this change: no problems whatsoever, smooth sailing all the way to a perfect XSD from any TYPO3-based ViewHelper collection.
  4. Claus Due commented on patch set 1 05/11/17, 15:06

    If you do test this, please be aware the generateschema command needs RenderingContext from TYPO3 to be specified as argument, or VH classes won't load with ObjectManager and thus won't have the ReflectionService => fatal errors.
  5. Elmar Hinz commented on patch set 1 05/11/17, 15:11

    You can always do something like this in the outermost method of a library. public function __construct($cache = null) { if($cache) { $this->cache = $cache; } else { $this->cache = $this->getDefaultCache(); } - no magic in the depth - independent from TYPO3 by cache injection - any reasonable default solution
  6. Claus Due commented on patch set 1 05/11/17, 15:20

    > You can always do something like this in the outermost method of a library. I'm sorry Elmar but this misses the point. The XSD generator is universal and it will not initialise caches for opinionated frameworks which implement it, just to be able to call prepareArguments() on the classes provided by said framework. But don't hesitate to go ahead and actually test this patch along with the schema generator. Take a look into the source of it, evaluate it against what Fluid normally would expect in terms of extracting ArgumentDefinitions and then soul-search a bit if all those requirements we impose are really rational.
  7. Elmar Hinz commented on patch set 1 05/12/17, 02:48

    The point is, that flaws in the design of libraries, should be addressed by improving the design, not by hacks to hide the flaws. https://github.com/auchenberg/volkswagen
  8. Claus Due commented on patch set 1 05/12/17, 11:04

    Try to understand that this isn't a matter of hiding an error. It's a matter of making that error never happen. Hiding an error means the error still exists. Making sure it does not happen makes the error not exist anymore. Do you understand?
  9. Claus Due commented on patch set 1 05/12/17, 11:05

    Also, the flaw in application design is in TYPO3, not in the application I'm trying to write which must use TYPO3 classes, but TYPO3 made it close to impossible to do so without faking a bootstrapping; a method I simply do not have available to me here.
  10. Elmar Hinz commented on patch set 1 05/12/17, 11:22

    > Also, the flaw in application design is in TYPO3, not in the > application I'm trying to write which must use TYPO3 classes, but > TYPO3 made it close to impossible to do so without faking a > bootstrapping; a method I simply do not have available to me here. Let's address the flaws: - Solved: Fluid is independent from extbase? - Flaw: View helpers have a forced dependency on extbase, which is only needed for some? - Flaw: Extbase lacks the option to inject caches independent form cache manager? - Flaw: Extbase lacks the option to inject a cache manager? Right, wrong, more?
  11. Christian Kuhn commented on patch set 1 05/12/17, 11:37

    Sorry, but still no: You are writing an application that generates stuff for view helpers based on typo3 fluid standalone. That's great. TYPO3 specific fluid view helpers however have a dependency to the objectmanager, extbase and thus: the TYPO3 core. Your xsd generator does have a dependency to the core if you want to create TYPO3 specific view helper addons. You should not hack around this fact. I'd suggest to create a second version of the generator: The base version has no TYPO3 dependency whatsoever and can be happily used with fluid standalone. The second one - a typo3 specific flavor of your project - depends on your base project AND the typo3 core, does properly bootstrap what is needed of the core and generates the TYPO3 specific stuff.
  12. Claus Due commented on patch set 1 05/12/17, 14:58

    I'm not "hacking round" any facts. I'm trying to explain that merely instanciating a ViewHelper class is unreasonably intertwined with the framework. I'll be as concrete as possible: I need to be able to call "prepareArguments" on a VH class instance. This function, in the normal implementation, uses a single class (ArgumentDefinition) which has no requirements for DI and so on. But TYPO3 ViewHelpers require the ReflectionService because TYPO3 overrides the prepareArguments method and involves Reflection. Alright - so, in order to merely generate a list of arguments a VH class supports, TYPO3 now adds the requirement of two different caches. Mind you: I'm not in any way intending to *render* the ViewHelpers. All I want is for them to be able to tell me which arguments they support. Should that really require a full TYPO3 context, honestly? The following solutions are possible then: 1. Remove render() method argument support in TYPO3 ViewHelpers, thus removing the ReflectionService and making classes able to deliver ArgumentDefinitions without reflection, and without object caches. 2. Implement the fallback suggested in this merge request, thus making the ReflectionService able to analyse classes without having caches configured; which works perfectly if we just fall back to a compatible cache. 3. Create yet another TYPO3 CMS only invention/utility which only works with TYPO3 CMS because TYPO3 CMS ViewHelpers are not compatible with Fluid unless TYPO3 framework is initialised. You'd recommend the third solution, which would be completely unnecessary if any of the other two were chosen. Now, we are already planning to implement the first solution, but it's not going to be ready soon and it's a breaking change. So, I suggest the second solution - this patch - as a solution to this and other problems like it that might arise in the future. I'm sorry to say that your proposed solution is imho the worst possible solution we could choose. We'd be simply insisting that *our* ViewHelper inventions are so tightly connected to *our* framework that not even analysing the argument definitions can be done without first initialising TYPO3 CMS. Note: this applies to other cases as well, not just ViewHelper schemas. I'm using this as a concrete example but other similar use cases would *also* be solved by implementing this patch. And can we please don't forget that what we're really discussing here is whether or not to throw an exception or provide a fully compatible fallback. It's that small of a change, I just can't fathom why you'd be so extremely opposed to the idea of that. Not in thise case and not in other cases like it. I just don't understand the logic here.
  13. Elmar Hinz commented on patch set 1 05/12/17, 15:23

    IMHO decoupling should be done in AbstractViewHelper, a much more outside level than hacking the cache. This is one of the annoyances of inheritance compared to composition. While it is easy to inject other objects, it's difficult to inject other parent classes. There is a method injectReflectionService. Inject a kind of dummy service and decoupling is done.
  14. Claus Due commented on patch set 1 05/12/17, 16:43

    Yes, the long term plan is to deprecate and remove those problems in AbstractViewHelper. But this doesn't solve the problem here! Yes, the service can be injected - but then the VH class requires special init and we're back at square one. This is exactly the thing I'm trying to avoid and why I'm arguing we adopt graceful fallbacks instead of falling flat on our faces as soon as you touch any Extbase-related class. 16 lines of changed core here, versus requiring a completely separate tool to build XSD files just because we throw an exception instead of falling back to a 100% compatible zero-config solution. I can't believe we've wasted so much time talking about this but I guess it's true: the simpler the change, the more arguing there has to be about it.
  15. Elmar Hinz commented on patch set 1 05/12/17, 17:34

    > Yes, the long term plan is to deprecate and remove those problems > in AbstractViewHelper. But this doesn't solve the problem here! > > Yes, the service can be injected - but then the VH class requires > special init and we're back at square one. This is exactly the > thing I'm trying to avoid and why I'm arguing we adopt graceful > fallbacks instead of falling flat on our faces as soon as you touch > any Extbase-related class. > > 16 lines of changed core here, versus requiring a completely > separate tool to build XSD files just because we throw an exception > instead of falling back to a 100% compatible zero-config solution. > I can't believe we've wasted so much time talking about this but I > guess it's true: the simpler the change, the more arguing there has > to be about it. Even if it is a few lines of code, it's the clean design of an application: * Select the classes YOU want to use, instead of relying on what THEY choose for you. * Wire classes up by injection. * Run. * Catch exceptions and handle them as it best suits your application context. I think we speak about an application, about the usage of TYPO3 classes outside the default application, with it's bootstrapping an DB dependencies.
  16. Claus Due commented on patch set 1 05/12/17, 17:50

    What's cleaner: - Throwing an fatal exception of a TYPO3-only type when metadata configuration does not exist and demanding that the configuration be added. - Falling back to a fully compatible zero-conf solution without preventing the manual configuration from being applied. What's simpler: - Demanding that yet another TYPO3 exclusive application be written which demands full TYPO3 bootstrapping to be able to do something that a simple composer dependency can do for you with any other ViewHelper than TYPO3 ones. - Gracefully falling back to give *just* enough transient memory resident zero-config implementations that the classes we provide will transparently allow the exact same docs extraction as any other VH class. So do we pick the nasty solution of throwing an exception and demand that TYPO3 classes only ever be loadable if TYPO3 is completely bootstrapped - or do we be just a TINY degree of flexibility in NOT demanding all of that? Why are we even thinking about this?!
  17. Elmar Hinz commented on patch set 1 05/12/17, 17:59

    > What's cleaner: > > - Throwing an fatal exception of a TYPO3-only type when metadata > configuration does not exist and demanding that the configuration > be added. > - Falling back to a fully compatible zero-conf solution without > preventing the manual configuration from being applied. > > What's simpler: > > - Demanding that yet another TYPO3 exclusive application be written > which demands full TYPO3 bootstrapping to be able to do something > that a simple composer dependency can do for you with any other > ViewHelper than TYPO3 ones. > - Gracefully falling back to give *just* enough transient memory > resident zero-config implementations that the classes we provide > will transparently allow the exact same docs extraction as any > other VH class. > > So do we pick the nasty solution of throwing an exception and > demand that TYPO3 classes only ever be loadable if TYPO3 is > completely bootstrapped - or do we be just a TINY degree of > flexibility in NOT demanding all of that? Why are we even thinking > about this?! An application that creates any kind of a cache, without needing a cache at all, can't be a clean solution.
  18. Claus Due commented on patch set 1 05/12/17, 18:13

    > An application that creates any kind of a cache, without needing a cache at all, can't be a clean solution. It demands a cache and that's the problem. It demands this, but normal such implementations do not. And the reason it demands a cache is horrible performance due to the demand of using reflection. It's just wrong from one end to the other.
  19. Elmar Hinz commented on patch set 1 05/12/17, 18:33

    > > An application that creates any kind of a cache, without needing > a cache at all, can't be a clean solution. > > It demands a cache and that's the problem. It demands this, but > normal such implementations do not. And the reason it demands a > cache is horrible performance due to the demand of using > reflection. It's just wrong from one end to the other. You get rid of the reflection by injection of a dummy reflection service, as long as parent::prepareArguments() is all you want.
  20. Claus Due commented on patch set 1 05/12/17, 18:37

    > You get rid of the reflection by injection of a dummy reflection service, as long as parent::prepareArguments() is all you want. And you can't do that without special handling of said ViewHelpers; handling which does not belong in an universal XSD generator. So we're back to square one: because of this requirement the only option is a TYPO3-specific XSD generator. Is this *really* what we want to demand? I can't believe it is...
  21. Elmar Hinz commented on patch set 1 05/12/17, 19:37

    > And you can't do that without special handling of said ViewHelpers; They all implement the same interface and inherit from the same abstract class.
  22. Elmar Hinz commented on patch set 1 05/13/17, 07:01

    > > You get rid of the reflection by injection of a dummy reflection > service, as long as parent::prepareArguments() is all you want. > > And you can't do that without special handling of said ViewHelpers; > handling which does not belong in an universal XSD generator. So > we're back to square one: because of this requirement the only > option is a TYPO3-specific XSD generator. > > Is this *really* what we want to demand? I can't believe it is... Sure you need an if condition, to check if you need a dummy service. It does not require a TYPO3-specific application.
  23. Claus Due commented on patch set 1 05/13/17, 09:16

    > Sure you need an if condition, to check if you need a dummy service. It does not require a TYPO3-specific application. Please think a bit about this you're saying here. You want a non-TYPO3-dependent library to contain a fake version of the ReflectionService from TYPO3 so it can inject this into TYPO3 ViewHelpers because they (as opposed to normal ones) require this special class. Now you've introduced either a hard dependency on TYPO3 and created a TYPO3-only application. Or you've introduced an invisible dependency through an invalid class that can only be operated if TYPO3 is installed alongside the XSD generator. Please acknowledge that neither of these is acceptable.
  24. Elmar Hinz commented on patch set 1 05/13/17, 09:34

    > > Sure you need an if condition, to check if you need a dummy > service. It does not require a TYPO3-specific application. > > Please think a bit about this you're saying here. > > You want a non-TYPO3-dependent library to contain a fake version of > the ReflectionService from TYPO3 so it can inject this into TYPO3 > ViewHelpers because they (as opposed to normal ones) require this > special class. > > Now you've introduced either a hard dependency on TYPO3 and created > a TYPO3-only application. > > Or you've introduced an invisible dependency through an invalid > class that can only be operated if TYPO3 is installed alongside the > XSD generator. > > Please acknowledge that neither of these is acceptable. I decoupled the application from the caches. That is definitely a progress compared to your solution. I could go even further. Your solution can't.
  25. Claus Due abandoned the change 05/13/17, 10:10

    I give up