Review main Merged

51405[FEATURE] Option to disable validation of stored records

Thomas Hohn CR +2 V +2 Normal +62 −0 updated 02/07/17, 21:20

Commit message

Adds two new configuration options to Install Tool which are used when
saving records using the DataHandler:

* disable the validation of records after saving them
* if validating, make a strict value comparison ('' !== 0) instead of
  the default loose comparison ('' == 0)

Review

  • Code review CR +2
    • Frans Saris +2
    • Claus Due +1
    • Christian Kuhn +1
  • Verified V +2
    • Frans Saris +2
    • Christian Kuhn +1
    • TYPO3com +1 CI

Files 3

Review log 6179

Show 4154 earlier messages
  1. Thomas Hohn uploaded patch set 1 01/24/17, 13:12

  2. Oliver Klee commented on patch set 2 01/24/17, 13:13

    Published edit on patch set 1.
  3. TYPO3com voted on patch set 1 V +1 01/24/17, 13:18 Completed build in 06m 22s on Tue, 24 Jan, 02:18 PM Test Summary: 55161 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6964

  4. Claus Due voted on patch set 2 CR +1 01/24/17, 13:19

  5. TYPO3com voted on patch set 2 V +1 01/24/17, 13:19 Completed build in 05m 52s on Tue, 24 Jan, 02:19 PM Test Summary: 55161 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6965

  6. Oliver Hader commented on patch set 2 CR -1 01/24/17, 14:55

    Imagine TCA defines a select field which might contain a CSV value like "10,20,30" - but the SQL definition just has an integer type defined for this field. How would you determine, that just "10" (as integer) is persisted? Next scenario: Writing 40 bytes to a field that is defined as varchar(32). If you think, this could be done by constraints, triggers or whatever, I think it should be part of this patch, shouldn't it?
  7. Thomas Hohn commented on patch set 2 01/24/17, 15:09

    In case 2 I would mean that the insert would have failed with an DB error. Case 1 you are right - the question is however if the enduser has access to this "info" and will notice that? An alternative could be to have a configurable flag determining if the if the checkStoredRecord should be executed or not.
  8. Thomas Hohn commented on patch set 2 01/24/17, 15:24

    In mysql for instance i can't insert a list '10,20.30' into an integer value..
  9. Thomas Hohn commented on patch set 2 01/25/17, 06:25

    It see the problem if you don't run the database in strict mode. Would it be possible to have some kind of configuration that could disable the check? We have experienced that this additional check is expensive when copying existing sub-trees in the backend.
  10. Oliver Hader commented on patch set 2 01/25/17, 11:24

    "Strict Mode" might be different on different DBMS and I don't think that all misconfigurations are caught by that. As an alternative I see a schema validator in the Install Tool that verifies a configured TCA type really can be persisted for the defined table schema.
  11. Claus Due commented on patch set 2 01/25/17, 12:00

    My opinion: the most damning argument against this is two parts - one, it is mostly concerned with catching problems caused by user inputting bad data (it resides in DataHandler which triggers only when you edit with FormEngine), and two, it runs consistently for any record being saved even when this saving is caused for example by copying a record. To start with number two: it is quite redundant to confirm that records copied from an existing record, or workspace placeholders, or automatically created versions, or even moving of existing records, can be persisted correctly. The likelihood it won't is extremely low (requires various "append/prepend" operations or hooks which cause data incompatibility). And the first one: given that the intention with this is to merely report whether user-input data was stored correctly, is it not more appropriate to *prevent* invalid values in the UX by restricting field length and so on? And don't we already do that with a wide range of constriction settings for pretty much all field types? Re: TCA/schema validator in install tool I'm fine with that, but it is beyond the scope of this change.
  12. Thomas Hohn commented on patch set 2 01/25/17, 12:25

    Could a possible middleway solution be to add an possibility to skip the check? I could live with that and as default the check would be enabled?
  13. Thomas Hohn uploaded patch set 3 01/26/17, 12:40

  14. TYPO3com voted on patch set 3 V -1 01/26/17, 12:40 Completed build in 06s on Thu, 26 Jan, 01:40 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7055

  15. Thomas Hohn uploaded patch set 4 01/26/17, 12:42

  16. TYPO3com voted on patch set 4 V +1 01/26/17, 12:47 Completed build in 04m 59s on Thu, 26 Jan, 01:47 PM Test Summary: 55149 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7056

  17. Claus Due commented on patch set 4 CR -1 01/28/17, 13:00

    Lots of unrelated whitespace changes in DataHandler - actual change looks good but the whitespace should definitely not be changed (if you want to do that, it's welcome, but should be a separate merge request).

    DefaultConfiguration.php:778

    I would rename this to simply "loose", the path to the setting already contains "checkStoredRecords".
  18. Thomas Hohn commented on patch set 4 01/28/17, 14:23

    Okay - seems unintentional. Easiest way ist to take patch set 3 and then reapply the changes or?
  19. Thomas Hohn uploaded patch set 5 01/28/17, 19:20

  20. TYPO3com voted on patch set 5 V +1 01/28/17, 19:27 Completed build in 07m 01s on Sat, 28 Jan, 08:27 PM Test Summary: 55619 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7163

  21. Thomas Hohn uploaded patch set 6 01/29/17, 06:54

  22. TYPO3com voted on patch set 6 V +1 01/29/17, 07:00 Completed build in 06m 10s on Sun, 29 Jan, 08:00 AM Test Summary: 55619 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7173

  23. Thomas Hohn uploaded patch set 7 01/29/17, 07:00

  24. TYPO3com voted on patch set 7 V +1 01/29/17, 07:06 Completed build in 05m 27s on Sun, 29 Jan, 08:06 AM Test Summary: 55619 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7174

  25. Stephan Großberndt commented on patch set 8 02/06/17, 09:23

    Published edit on patch set 7.
  26. Georg Ringer commented on patch set 8 CR -1 02/06/17, 09:26

    DefaultConfiguration.php:778

    so what does loose do?
  27. Stephan Großberndt commented on patch set 8 CR -1 02/06/17, 09:27

    DataHandler.php:6836

    twice "if ($this->enableLogging) {"

    DefaultConfiguration.php:776

    Add comment strings like in the statements above to show these settings in the install tool.

    DefaultConfiguration.php:778

    'loose' is not mentioned in the docs yet.

    Important-79438-DisableCheckStoredRecordInDatahandler.rst:4

    The patch is prefixed as [FEATURE], name of this file and title have to be set accordingly
  28. TYPO3com voted on patch set 8 V +1 02/06/17, 09:29 Completed build in 05m 49s on Mon, 6 Feb, 10:29 AM Test Summary: 55955 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7550

  29. Stephan Großberndt commented on patch set 8 02/06/17, 09:31

    I am not convinced of the patch in the current state. Not setting the proper title for a copied element seems like introducing a bug with this feature to me. The option to disable 'checkStoredRecord' was there in DataHandler before, but only usable by code instantiating DataHandler itself and then for the single run. Are you really sure this cannot create havoc? Claus mentioning hooks - if a hook prepends/appends data on copy (translate/whatever) the data might not fit any more.
  30. Thomas Hohn commented on patch set 8 02/06/17, 09:36

    It's only the title in the sys_log entry - I can live with that if it gives me a speed up of a factor 2 when copying big sub-trees. I'll fix the other proposals/issues
  31. Stephan Großberndt commented on patch set 8 02/06/17, 09:54

    Ok, the description need to state that more clearly then. maybe the log message could be changed accordingly too then?
  32. Tymoteusz Motylewski commented on patch set 8 CR -1 02/06/17, 10:15

    DataHandler.php:103

    the patch is also breaking, as it's removing public properties. Please add correct rst file stating it.

    Important-79438-DisableCheckStoredRecordInDatahandler.rst:27

    I don't understand this sentence. "displayed" - where? "uid and table are still provided" - where? why?
  33. Tymoteusz Motylewski commented on patch set 8 02/06/17, 10:22

    This patch is removing properties and make datahandler rely on Global setting. Shouldn't it be the other way around? I mean that some property is initialized by globals and then used in the code? In general I tend to agree with Claus that the check is redundant. What are the real life usecases where the check will discover some differences? Maybe we can drop the check completely after some investigation or adding validation somewhere else?
  34. Thomas Hohn commented on patch set 8 02/06/17, 10:27

    Hi Tymoteusz - so keeping the public variables and initialise them in the construct based on the content of the globals - thus making it possible to overwrite - this would make the change non-breaking :-)
  35. Thomas Hohn uploaded patch set 9 02/06/17, 11:56

  36. TYPO3com voted on patch set 9 V +1 02/06/17, 12:05 Completed build in 08m 40s on Mon, 6 Feb, 01:05 PM Test Summary: 55955 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7570

  37. Thomas Hohn commented on patch set 9 02/07/17, 05:01

    Would be glad for a couple of +1 from the reviewers after my changes ;-)
  38. Thomas Hohn uploaded patch set 10 02/07/17, 09:54

  39. Claus Due commented on patch set 10 CR -1 02/07/17, 09:58

    RST needs to be updated regarding the `loose` setting, and it should be mentioned in the RST what the setting does (a bit more than "sets checkStoredRecords_loose in DataHandler should be explained).
  40. TYPO3com voted on patch set 10 V +1 02/07/17, 10:00 Completed build in 05m 37s on Tue, 7 Feb, 11:00 AM Test Summary: 56139 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7661

  41. Claus Due commented on patch set 10 02/07/17, 10:02

    Feature-79438-DisableCheckStoredRecordInDatahandler.rst:21

    Uses the old setting name. Add something like "causes loose comparisons to be made in checks, i.e. `'1' == 1` instead of `'1' === 1`".
  42. Claus Due commented on patch set 10 02/07/17, 10:02

    Uses old setting name.
  43. Thomas Hohn uploaded patch set 11 02/07/17, 10:39

  44. Claus Due voted on patch set 11 CR +1 02/07/17, 10:41

  45. TYPO3com voted on patch set 11 V +1 02/07/17, 10:45 Completed build in 05m 39s on Tue, 7 Feb, 11:45 AM Test Summary: 56139 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7665

  46. Stephan Großberndt commented on patch set 12 02/07/17, 12:56

    Published edit on patch set 11.
  47. TYPO3com voted on patch set 12 V +1 02/07/17, 13:02 Completed build in 05m 38s on Tue, 7 Feb, 02:02 PM Test Summary: 56139 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7678

  48. Stephan Großberndt uploaded patch set 13 02/07/17, 13:08

  49. Stephan Großberndt voted on patch set 13 CR +1 02/07/17, 13:08

  50. Stephan Großberndt voted on patch set 13 CR 0 V -1 02/07/17, 13:14

  51. TYPO3com voted on patch set 13 V +1 02/07/17, 13:15 Completed build in 07m 23s on Tue, 7 Feb, 02:15 PM Test Summary: 56139 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7681

  52. Stephan Großberndt uploaded patch set 14 02/07/17, 13:21

  53. Stephan Großberndt voted on patch set 14 CR +1 V +1 02/07/17, 13:22

  54. Stephan Großberndt uploaded patch set 15 02/07/17, 13:26

  1. Stephan Großberndt voted on patch set 15 CR +1 V +1 02/07/17, 13:26

  2. TYPO3com voted on patch set 14 V +1 02/07/17, 13:27 Completed build in 05m 50s on Tue, 7 Feb, 02:27 PM Test Summary: 56139 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7683

  3. Thomas Hohn voted on patch set 15 CR +1 02/07/17, 13:30

  4. TYPO3com voted on patch set 15 V -1 02/07/17, 13:33 Completed build in 06m 47s on Tue, 7 Feb, 02:33 PM Test Summary: 1 of 56139 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7685

  5. Thomas Hohn commented on patch set 15 V +1 02/07/17, 14:10

    Also testet latest version - io :)
  6. Frans Saris commented on patch set 15 02/07/17, 18:25

    DataHandler.php:6993

    Is this related to the feature?
  7. Thomas Hohn commented on patch set 15 02/07/17, 18:34

    DataHandler.php:6993 Reply

    Hi - yes and no - instead of using an own query we use a standard method for a standard query - which in my opinion makes good sense :-)
  8. Frans Saris commented on patch set 15 02/07/17, 18:56

    How to test this? can't we create some functional tests that make sure this does what is should do?
  9. TYPO3com voted on patch set 15 V +1 02/07/17, 19:01 Completed build in 5h 41m 22s on Tue, 7 Feb, 08:01 PM Test Summary: 56139 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7685

  10. Thomas Hohn commented on patch set 15 02/07/17, 19:12

    Maybe - but I'm sort of "running" out of time. Easiest way to test: Create a page copy the page and check sys_log. Set the value to false copy the page again - should give same records entries except that when set to false - the title of the page in sys_log ONLY would be [No title] :-)
  11. Christian Kuhn commented on patch set 15 CR -1 02/07/17, 19:21

    DataHandler.php:710

    i'm not a huge fan of this: to use the feature, you need to change $globals['typo3-conf-vars'], but that array should be read-only after it has been compiled by bootstrap during runtime.
  12. Thomas Hohn commented on patch set 15 02/07/17, 19:23

    Christian - then please tell me where to store a config like that :-)
  13. Thomas Hohn commented on patch set 15 02/07/17, 19:26

    Maybe I misunderstod you - the intention is either to have this feature enabled always or never perform the check
  14. Christian Kuhn commented on patch set 15 02/07/17, 19:41

    well, i'd probably keep the public properties and just set them to false in those scopes where i really want that - for instance by hanging into ext:impexp somehow and disabling those. do you really want to disable this validation globally for a full instance and take the risk that something may not end up in db correctly without telling anyone?
  15. Claus Due commented on patch set 15 CR -1 02/07/17, 19:48

    DataHandler.php:710 Reply

    I believe the intention is to override this either in a "site extension" or using AdditionalConfiguration. So, not intended to be changed on-the-fly. The properties are still public and can be changed from, for example, DataHandler pre-hooks - the TYPO3_CONF_VARS only serve as a global default :)

    DataHandler.php:6993 Reply

    Technically it isn't part of the change so to increase the chance it gets merged, please revert this part.
  16. Thomas Hohn commented on patch set 15 02/07/17, 19:50

    We run MySql in strict mode and the DB layer together with validation in the backend should ensure that an SQL error is thrown long before that this check is running - so the short answer is YES
  17. Thomas Hohn uploaded patch set 16 02/07/17, 19:56

  18. TYPO3com voted on patch set 16 V +1 02/07/17, 20:04 Completed build in 07m 43s on Tue, 7 Feb, 09:03 PM Test Summary: 56147 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7734

  19. Thomas Hohn uploaded patch set 17 02/07/17, 20:32

  20. TYPO3com voted on patch set 17 V +1 02/07/17, 20:41 Completed build in 08m 36s on Tue, 7 Feb, 09:41 PM Test Summary: 56195 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7742

  21. Claus Due commented on patch set 17 CR +1 02/07/17, 21:02

    Now THIS is a patch that I like to see! 15 times as much documentation as code changed :) I haven't verified this due to local mess, but just by reading, this should work exactly as advertised.
  22. Frans Saris voted on patch set 17 CR +1 02/07/17, 21:16

  23. Christian Kuhn commented on patch set 17 CR +1 V +1 02/07/17, 21:17

    ok. fine. get it in.
  24. Frans Saris voted on patch set 17 CR +2 V +2 02/07/17, 21:19

  25. Gerrit merged patch set 17 02/07/17, 21:20