Review main Merged

64652[BUGFIX] Ensure uniqid calls use more_entropy

Tizian Schmidlin CR +2 V +2 Normal +38 −22 updated 09/04/20, 09:08

Commit message

Precision of `uniqid()` on Windows systems without passing
`$more_entropy=true` has only single-second-resolution which will
lead to non-unique ids on subsequent calls.

In order to mitigate this issue TYPO3 provides the function
`StringUtility::getUniqueId($prefix = '')` which calls
`uniqid()` with parameter `$more_entropy` always set to true.

Using `uniqid()`, especially with `$more_entropy` set to true,
is quite slow, but for the purposes TYPO3 is using it
(i.e. creating unique field names for backend forms, path
identifiers in some modules etc.) it is good enough and
another solution would not provide any measurable benefit.

Review

  • Code review CR +2 Done
    • Christian Kuhn +2
    • Oli Bartsch +1
    • Alexander Schnitzler +1
    • Jörg Bösche +1
  • Verified V +2 Done
    • Christian Kuhn +2
    • Oli Bartsch +1
    • Alexander Schnitzler +1
    • Jörg Bösche +1
    • TYPO3com +1 CI

Issues 1

Files 18

Review log 4257

Show 2432 earlier messages
  1. Tizian Schmidlin uploaded patch set 1 06/02/20, 07:39

  2. TYPO3com voted on patch set 1 V -1 06/02/20, 07:39 Completed build in 24s on Tue, 2 Jun, 07:39 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3737

  3. Tizian Schmidlin uploaded patch set 2 06/02/20, 08:02

  4. Tizian Schmidlin uploaded patch set 3 06/02/20, 08:03

  5. TYPO3com commented on patch set 2 06/02/20, 08:03 Completed build in 24s on Tue, 2 Jun, 08:03 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3739

  6. TYPO3com voted on patch set 3 V +1 06/02/20, 08:14 Completed build in 11m 10s on Tue, 2 Jun, 08:14 AM Test Summary: 72840 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3740

  7. Stephan Großberndt commented on patch set 3 06/02/20, 20:14

    StringUtility::getUniqueId() should be used here. But you stumbled upon a bigger problem. There are still several places in the core where uniqid() without TRUE is used. Once upon a time we touched that topic already: https://review.typo3.org/c/Packages/TYPO3.CMS/+/30948/
  8. Tizian Schmidlin commented on patch set 3 06/03/20, 06:56

    > Patch Set 3: > > StringUtility::getUniqueId() should be used here. > But you stumbled upon a bigger problem. There are still several places in the core where uniqid() without TRUE is used. > Once upon a time we touched that topic already: https://review.typo3.org/c/Packages/TYPO3.CMS/+/30948/ I also discovered this issue around that time on an old system and had to patch it by hand at the time... Wasn't funny. I'll take a look at StringUtility::getUniqueId() and see if I can expand the patch to all instances in the core.
  9. Tizian Schmidlin uploaded patch set 4 06/03/20, 07:14

  10. TYPO3com voted on patch set 4 V -1 06/03/20, 07:26 Completed build in 11m 39s on Wed, 3 Jun, 07:26 AM Test Summary: 72842 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3757

  11. Tizian Schmidlin uploaded patch set 5 06/03/20, 07:40

  12. Tizian Schmidlin uploaded patch set 6 06/03/20, 07:46

  13. TYPO3com commented on patch set 5 06/03/20, 07:52 Completed build in 12m 12s on Wed, 3 Jun, 07:52 AM Test Summary: 72842 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3759

  14. TYPO3com voted on patch set 6 V +1 06/03/20, 08:02 Completed build in 15m 29s on Wed, 3 Jun, 08:02 AM Test Summary: 72842 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3760

  15. Stephan Großberndt commented on patch set 6 06/03/20, 08:07

    SiteConfigurationController.php:368

    (string)random_int(0, mt_getrandmax()) is a case of "I want it VERY unique!!1" I guess? :) We should discuss (with core members) whether this is actually needed, I'd rather simply go for $identifier .= StringUtility::getUniqueId('-');

    BasicFileUtility.php:128

    Remove empty string parameter

    SvgIconProviderTest.php:83

    Using StringUtility::getUniqueId at all is overkill here as tempnam() already creates a unique filename, Should be $testFile = GeneralUtility::tempnam('svg_', '.svg');

    SvgIconProviderTest.php:87

    $this->testFilesToDelete[] = GeneralUtility::tempnam('svg_', '.svg');

    AjaxWidgetContextHolder.php:96

    $ajaxWidgetId = md5(StringUtility::getUniqueId());

    DatePickerViewHelper.php:98

    $id = 'field' . md5(StringUtility::getUniqueId());

    UpgradeController.php:742

    getUniqueId already removes the dots and does not need "moar" randomness $preparedHit['uniqueId'] = StringUtility::getUniqueId();

    UpgradeController.php:771

    $parsedRestFile['uniqueId'] = StringUtility::getUniqueId();

    PreviewUriBuilder.php:217

    $keyword = md5(StringUtility::getUniqueId());
  16. Benni Mack commented on patch set 6 06/03/20, 08:10

    nice patch

    BasicFileUtility.php:128

    empty '' can be ommited.

    Typo3tempFileServiceTest.php:45

    wouldn't do this in tests.
  17. Tizian Schmidlin uploaded patch set 7 06/03/20, 08:36

  18. Stephan Großberndt updated the commit message in patch set 8 06/03/20, 08:45

  19. Tizian Schmidlin commented on patch set 8 06/03/20, 08:53

    > Patch Set 6: > > (2 comments) > > nice patch Thanks, I included the suggestions made by Stephan and you.

    SiteConfigurationController.php:368 Reply

    That's what I thought, too, when I replaced it, but as you say, this might be something at least another core member would have to back before we change it. I'll go with it, though.

    BasicFileUtility.php:128 Reply

    Ack

    BasicFileUtility.php:128 Reply

    Done

    SvgIconProviderTest.php:83 Reply

    Ack

    SvgIconProviderTest.php:87 Reply

    Ack

    AjaxWidgetContextHolder.php:96 Reply

    Ack

    DatePickerViewHelper.php:98 Reply

    Done

    UpgradeController.php:742 Reply

    Ack

    UpgradeController.php:771 Reply

    Ack

    Typo3tempFileServiceTest.php:45 Reply Unresolved

    So you would rather use a static name for the directory?

    PreviewUriBuilder.php:217 Reply Unresolved

    So no prefix anymore?
  20. Stephan Großberndt commented on patch set 8 06/03/20, 08:54

    SvgIconProviderTest.php:82

    this has to be separated with a comma instead as thats a second parameter for the suffix $testFile = GeneralUtility::tempnam('svg_', '.svg');

    SvgIconProviderTest.php:86

    same

    PreviewUriBuilder.php:217

    remove microtime()
  21. Benni Mack commented on patch set 8 06/03/20, 08:55

    PreviewUriBuilder.php:217 Reply

    yes. had the same feeling here :)
  22. Stephan Großberndt commented on patch set 8 06/03/20, 09:04

    Typo3tempFileServiceTest.php:44

    This is wrong now. This call will create a file with that name which is not intended (and may or should break stuff). I didn't understand Bennis comment either though. @benni: did you mean to refrain from using StringUtility in tests at all? $this->directoryName = StringUtility::getUniqueId('test'); is correct in my opinion.
  23. Benni Mack commented on patch set 8 06/03/20, 09:05

    Typo3tempFileServiceTest.php:44 Reply

    yes, I don't see a benefit having more entropy in tests right now.
  24. Stephan Großberndt commented on patch set 8 06/03/20, 09:08

    Typo3tempFileServiceTest.php:44 Reply

    Question is: May this be called twice in a single second if tests run fast on a windows system so a collision may happen.. Like usage of GeneralUtility in tests to make sure everything works in a consistent way I think its ok to do that with getUniqueId() too.
  25. TYPO3com voted on patch set 7 V -1 06/03/20, 09:09 Completed build in 26m 18s on Wed, 3 Jun, 09:09 AM Test Summary: 6 of 73329 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3764

  26. Tizian Schmidlin uploaded patch set 9 06/03/20, 09:12

  27. TYPO3com voted on patch set 8 V -1 06/03/20, 09:17 Completed build in 18m 14s on Wed, 3 Jun, 09:17 AM Test Summary: 6 of 73329 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3765

  28. Stephan Großberndt commented on patch set 9 06/03/20, 09:20

    SvgIconProviderTest.php:83

    Here GeneralUtility::tempnam() was used in the original code and should be. otherwise the tests will fail too as no file will be created. please use $testFile = GeneralUtility::tempnam('svg_', '.svg'); which will prefix with 'svg_', add the suffix '.svg' and create that file

    SvgIconProviderTest.php:87

    same
  29. Tizian Schmidlin uploaded patch set 10 06/03/20, 09:23

  30. Stephan Großberndt commented on patch set 10 06/03/20, 09:25

    SvgIconProviderTest.php:83 Reply

    Done

    SvgIconProviderTest.php:87 Reply

    Done

    Typo3tempFileServiceTest.php:45

    getUinqueId should be getUniqueId
  31. Tizian Schmidlin uploaded patch set 11 06/03/20, 09:26

  32. Stephan Großberndt commented on patch set 11 06/03/20, 09:31

    thanks a lot for your work, Tizian.

    Typo3tempFileServiceTest.php:45 Reply

    Done
  1. TYPO3com voted on patch set 9 V -1 06/03/20, 09:31 Completed build in 17m 05s on Wed, 3 Jun, 09:31 AM Test Summary: 12 of 73329 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3767

  2. Stephan Großberndt voted on patch set 11 CR +1 06/03/20, 09:38

  3. TYPO3com voted on patch set 10 V -1 06/03/20, 09:38 Completed build in 14m 33s on Wed, 3 Jun, 09:38 AM Test Summary: 6 of 73329 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3768

  4. TYPO3com voted on patch set 11 V -1 06/03/20, 09:44 Completed build in 15m 58s on Wed, 3 Jun, 09:44 AM Test Summary: 1 of 73329 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3769

  5. TYPO3com voted on patch set 11 V +1 06/03/20, 10:13 Completed build in 1h 01m 06s on Wed, 3 Jun, 10:13 AM Test Summary: 72842 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3769

  6. Tomas Norre Mikkelsen commented on patch set 11 CR +1 06/05/20, 08:06

    From the code I looks ok, I'm not sure about the performance impact you are mentioning your self.
  7. Tizian Schmidlin commented on patch set 11 06/05/20, 08:40

    > Patch Set 11: Code-Review+1 > > From the code I looks ok, I'm not sure about the performance impact you are mentioning your self. The remark about it being slower stems from the php.net documentation page, a benchmark showed that it was 0.2% slower to use $more_entropy than to not use it, which amounts to 20ms over 100'000 calls, I think TYPO3 can handle that.

    SiteConfigurationController.php:368 Reply

    Done

    SvgIconProviderTest.php:82 Reply

    Ack

    SvgIconProviderTest.php:83 Reply Unresolved

    *facepalm*

    SvgIconProviderTest.php:86 Reply

    Ack

    SvgIconProviderTest.php:87 Reply Unresolved

    *double-facepalm*

    Typo3tempFileServiceTest.php:44 Reply Unresolved

    --True, but seeing that this should be a temporary file name, isn't using tempnam the right way to do it?-- I've seen the errors of my ways and changed it back to getUniqueId()

    Typo3tempFileServiceTest.php:45 Reply

    Done

    PreviewUriBuilder.php:217 Reply

    Done
  8. Frank Nägler voted on patch set 11 CR +1 06/05/20, 10:30

  9. Tomas Norre Mikkelsen commented on patch set 11 06/05/20, 11:34

    > Patch Set 11: > > (8 comments) > > > Patch Set 11: Code-Review+1 > > > > From the code I looks ok, I'm not sure about the performance impact you are mentioning your self. > > The remark about it being slower stems from the php.net documentation page, a benchmark showed that it was 0.2% slower to use $more_entropy than to not use it, which amounts to 20ms over 100'000 calls, I think TYPO3 can handle that. I think so too.
  10. Andreas Kienast voted on patch set 11 V +1 CR +1 06/05/20, 11:55

  11. Jörg Bösche uploaded patch set 12 09/02/20, 09:08

  12. TYPO3com voted on patch set 12 V -1 09/02/20, 09:20 Completed build in 12m 02s on Wed, 2 Sep, 09:20 AM Test Summary: 72855 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5125

  13. TYPO3com voted on patch set 12 V +1 09/02/20, 09:27 Completed build in 31m 32s on Wed, 2 Sep, 09:27 AM Test Summary: 73164 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5125

  14. Jörg Bösche voted on patch set 12 V +1 CR +1 09/03/20, 10:12

  15. Christian Kuhn updated the commit message in patch set 13 09/03/20, 20:08

  16. Christian Kuhn commented on patch set 13 V +1 CR +1 09/03/20, 20:08

    I'm removing 9.5 here, hope that is not a critical issue.
  17. TYPO3com voted on patch set 13 V +1 09/03/20, 20:25 Completed build in 16m 48s on Thu, 3 Sep, 08:25 PM Test Summary: 73180 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-5203

  18. Jörg Bösche voted on patch set 13 V +1 CR +1 09/04/20, 06:03

  19. Alexander Schnitzler commented on patch set 13 CR -1 09/04/20, 08:10

    SvgIconProviderTest.php:82

    Why drop the uniqueness here?
  20. Tizian Schmidlin commented on patch set 13 09/04/20, 08:12

    SvgIconProviderTest.php:82 Reply

    tempnam already generates a uniq name and since we are in the scope of tests, this little change might speed up the whole process, too.
  21. Christian Kuhn commented on patch set 13 09/04/20, 08:38

    SvgIconProviderTest.php:82 Reply

    +1 Tizian
  22. Alexander Schnitzler commented on patch set 13 V +1 CR +1 09/04/20, 08:39

    SvgIconProviderTest.php:82 Reply

    Ack
  23. Oli Bartsch voted on patch set 13 V +1 CR +1 09/04/20, 08:49

  24. Christian Kuhn voted on patch set 13 V +2 CR +2 09/04/20, 09:08

  25. Christian Kuhn merged patch set 14 09/04/20, 09:08