Review main Abandoned

50559[FEATURE] Allow reference index updating to be disabled

Claus Due CR 0 V 0 Easy +32 −1 updated 12/04/16, 09:49

Commit message

This change adds a global configuration option which can
be used to disable the reference index updating that
normally happens when doing record operations.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 3

Review log 2225

  1. Claus Due uploaded patch set 1 11/09/16, 18:33

  2. Claus Due commented on patch set 1 11/09/16, 18:34

    On my test setup this toggle saves ~3,000 requests when copying a page tree with 25 pages and ~100 records without translations.
  3. TYPO3com voted on patch set 1 V -1 11/09/16, 18:38 Completed build in 04m 50s on Wed, 9 Nov, 07:38 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4735

  4. Claus Due uploaded patch set 2 11/09/16, 18:48

  5. TYPO3com voted on patch set 2 V -1 11/09/16, 18:57 Completed build in 08m 43s on Wed, 9 Nov, 07:57 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4737

  6. Claus Due uploaded patch set 3 11/09/16, 18:59

  7. TYPO3com voted on patch set 3 V +1 11/09/16, 19:08 Completed build in 08m 54s on Wed, 9 Nov, 08:08 PM Test Summary: 55265 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-4740

  8. Oliver Hader commented on patch set 3 CR -1 11/09/16, 21:46

    I don't think this is a good idea. There are several components in the core that expect a valid reference index (e.g. resolving child-elements to be published in workspace context or deleting a file that still is referenced).
  9. Markus Klein commented on patch set 3 CR -1 11/09/16, 22:06

    I'm with Olly here, albeit I'm not the biggest fan of this, it is a very welcome thing especially for soft references
  10. Claus Due commented on patch set 3 11/09/16, 23:00

    I know and respect the reason for having the reference index, but please consider the implication: I have a use case with copying just 25 pages, no translations, and around 100 content records. This results in a whopping 3,000+ SQL queries because DataHandler performs this update many, many types over. The toggle is definitely not intended for regular usage. It is a means to allow site administrators to say "do not perform reference indexing here - I will make sure it happens another way (one that doesn't impact BE performance this severely - such as CLI or manual SQL in cron)". For experts with a very real problem to solve! ;)
  11. Claus Due commented on patch set 3 11/09/16, 23:00

    s/types/times
  12. Markus Klein commented on patch set 3 11/10/16, 18:08

    Yes those queries are a lot, still the refindex is an integral part or the core and cause weird problems especially with FAL. Example: Add an image to RTE and have refindex disabled. Usage will not be counted and the file can be deleted in filelist even though it is in use. I guess we rather have to look into DataHandler here how we can reduce the fat amount of queries. Moreover the refindex update can be done at the very end of the copy action.
  13. Claus Due wrote 11/15/16, 15:58

    Topic changed from 78629 to performance
  14. Jigal van Hemert commented on patch set 3 CR -1 11/16/16, 19:14

    Bad idea, as pointed out by Markus and Oliver. It will soon be advised as an option to improve performance for free and when the first problems appear TYPO3 will be blamed. Performance improvements are fine as long as it won't break functionality that is needed.
  15. Claus Due commented on patch set 3 11/20/16, 21:29

    A couple of counter-arguments to address the arguments that 1) this must happen always when any record gets saved and 2) that people will, without knowing the reason behind, recommend it as a performance boost, while at the same time 3) forgetting to also explain that when disabled, reference indexing MUST be done manually and you risk XYZ integrity problems. Re 1), the reference index's main responsibility is to keep track of references in order to warn before deleting references and to cascade correctly when a record is deleted. This DOES mean there may arise inconsistency if a record is manipulated and an action which uses the reference index is triggered before the scheduled indexing runs (see 2). However: such inconsistency would be immediately fixed when the scheduled task runs. Note also here that any Extbase Repository intentionally disables reference indexing; which means the potential problems you face by enabling this for the BE, are the same potential problems you risk when manipulating Extbase records (which had relations attached). I have yet to find any case where this caused a fatal error. Re 2) I don't see this as any different from other performance improvement suggestions which change the normal TYPO3 behavior. Any action you take obviously has consequences, same as for example enabling the "don't respect no_cache" flag. Re 3) it will VERY quickly become known if disabling this causes any practical problems for sites. Literally it would only take a single stackoverflow question-and-answer to properly explain what it does, why you'd do it, and what you must also do if you use it. Finally: the system extension "lowlevel" contains an action to both test and update the reference index, an action which has documentation about how to enter it in crontab, and which clearly explains why you would run the task. So we *already* have a tool that does this task for us when (reason unregarded) it has not happened in the web interface. Among other things, this task is sensible to run when you exactly have Extbase repositories with objects using relations. This patch, plus the documentation for it, plus that existing crontab-useful lowlevel task, plus the documentation for that, is IMHO the perfect combination that allows expert users to squeeze more performance from the web interface and put more load on crontabs instead. Misuse of this might be blamed on TYPO3 but such blame would inevitably just reflect badly on the one blaming us for it. Since it is so clearly documented and comes with "experts only!" warnings attached. Hopefully these points address the arguments against it?
  16. Helmut Hummel commented on patch set 3 CR -2 12/03/16, 15:05

    OK, here is the third core team member to oppose this change, me :) I understand the performance concerns and I understand that something similar to "eventual consistency" might be appropriate for some cases. It is just nothing we are able to officially support with the current approach how to deal with data. This change is a hack to gain performance which is legitimate to introduce for a special project but should not be introduced and supported by the product officially. Since this is only one property to be changed, it is a fair deal to introduce an XCLASS to achieve what is requested here in the projects where the downsides of this hack are acceptable.
  17. Claus Due commented on patch set 3 12/03/16, 15:13

    Perhaps some numbers would be appropriate here - I still request that you reconsider this, as there are some tremendous performance issues with ReferenceIndex and RelationHandler. To illustrate: When you have a sys_category that gets used 10,000 times then the ReferenceIndex processes 10,000 relations every time you delete/copy a record that has a relation to this sys_category. For bulk operations this is compounded, e.g. deleting/copying two records causes 20,000 processing iterations. Making the problem worse is the fact that internally in RelationHandler (which is quite widely used also in FE) updating of the sys_refindex may *also* be triggered. ReferenceIndex itself also uses RelationHandler which causes recursive calling to update sys_refindex. The resulting performance issues are so severe that they cause timeouts >60s when copying just 2,000 user records. My client is very affected by this since they store users in a central storage and deploy them. Making the reference index updating asynchronous completely removes the bottleneck in all operations which import or delete any record which has a significant number of relations (sys_category is a great example here). And scheduling it to then be performed immediately after, for example using hooks, would be a viable solution. The reason I request this flag is exactly to avoid an XCLASS which replaces such a vital component of TYPO3 with one that then has to be maintained and be cross-version-compatible. This flag is such a tiny little change (literally 4 lines of code) and would allow seriously enterprise level setups to not be so drastically affected by reference indexing being so very inefficient. Side note: I've spent weeks now thinking about how to improve that reference indexing and each improvement is minor in comparison to making the indexing asynchronous. Please reconsider. If you need any more background information and numbers on this I am more than happy to supply them! ;)
  18. Helmut Hummel commented on patch set 3 12/03/16, 15:15

    > as there are some tremendous performance issues with ReferenceIndex and RelationHandler I don't doubt that. Lets fix them in a clean and clear way instead of closing the eyes and hoping for the best :)
  19. Claus Due commented on patch set 3 12/03/16, 15:17

    Another comment: I would accept any number of warnings in for example install tool or reports when this flag is enabled. I sincerely believe we can make this so visible to sysadmins that it becomes a completely informed decision to disable this.
  20. Helmut Hummel commented on patch set 3 12/03/16, 15:17

    > Making the reference index updating asynchronous This would require an architecture, where this is tackled automatically without any additional configuration. E.g. we could invest some time in research how to send out the web response and still do some more heavy lifting after that.
  21. Helmut Hummel commented on patch set 3 12/03/16, 15:19

    Another thing: not everything must be part of the core. As I pointed out, creating an extension that adds a (very safe and forward compatible) XCLASS would be a matter of minutes, which would be better spent than discussing integration into core (IMHO)
  22. Claus Due commented on patch set 3 12/03/16, 15:23

    > Another thing: not everything must be part of the core. Very true! But: something which can improve responsiveness during record operations by upwards of 90% reduction in wall time just might be worth having a bit of discussion about, rather than going straight to the EXT-to-override way (which as you know I am more than happy to do!)
  23. Claus Due commented on patch set 3 12/04/16, 00:34

    Regardless what happens to this merge request, there's now a community extension published and ready to use, which delegates updates to a queue that gets processed by a command controller. https://github.com/NamelessCoder/asynchronous_reference_indexing
  24. Anja Leichsenring commented on patch set 3 12/04/16, 08:10

    can we then abandon this one? It will gain no approval, from what I can see.
  25. Oliver Hader abandoned the change 12/04/16, 09:49

    Since four people now mentioned their concerns on disabling the reference index, I'm closing this change. It's more a candidate for a custom and local XCLASS instead of adding side-effects to the TYPO3 core. Thanks for your understanding...