Review main Abandoned

70246[BUGFIX] Fix ArrayUtility::isAssociative detection

Stephan Jorek CR 0 V 0 Normal +216 −5 updated 04/12/25, 13:17

Commit message

This commit changes the implementation of ArrayUtility::isAssociative
from checking if at least one key is a string, to checking if the array
is not empty and if not all keys are consecutively numbered. It uses
the array_is_list function introduced with PHP 8.1 or its stub provided
by symfony/polyfill-php81.

For consistency purposes the SingleFieldContainer::isAssociativeArray
implementation in the backend-extension uses the array_is_list function
too.

The current implementation of the ArrayUtility::isAssociative method,
assumes an array to be associative, when at least one key is a string.
So associative arrays with non-consecutively numbered keys are assumed
to be simple lists. But when merging YAML-configurations using imports
in Form-Framework the keys of associative arrays with non-consecutively
numbered keys must be preserved.

An array with non-consecutively numbered keys looks like:

  [
    10 => 'value',
    20 => 'value'
  ]

A simple list looks like:

  [
    'value',
    'value'
  ]

The later uses consecutively-numbered keys, whereas the former does not.

The current implementation leads to array-merges of associative arrays
with non-consecutively numbered keys, where all keys get renumbered in …

  ArrayUtility::replaceAndAppendScalarValuesRecursive($array1, $array2)

So in order merge two arrays like the following, with the above method, …

  $array1 = [
    10 => 'value 1',
    20 => 'value 2',
  ];

… and …

  $array2 = [
    20 => 'value 3',
    30 => 'value 4',
  ];

… into …

  $result = [
    10 => 'value 1',
    20 => 'value 3',
    30 => 'value 4',
  ];

… the implementation of ArrayUtility::isAssociative needs to be fixed.

These kind of merges are needed in the Form-Framework when using imports
in YAML-configuration, where the import adjusts existing keys in the
existing YAML-configuration. See both releated issues of the commit to
understand this in detail.

Review

  • Code review CR 0
  • Verified V 0

Issues 2

Files 3

Review log 4564

Show 2539 earlier messages
  1. Stephan Jorek uploaded patch set 1 08/06/21, 11:28

  2. Stephan Jorek updated the commit message in patch set 2 08/06/21, 11:35

  3. core-ci voted on patch set 1 V +1 08/06/21, 11:44 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14194

  4. Stephan Jorek uploaded patch set 3 08/06/21, 11:46

  5. Stephan Jorek updated the commit message in patch set 4 08/06/21, 11:47

  6. core-ci voted on patch set 2 V +1 08/06/21, 11:51 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14195

  7. Stephan Jorek updated the commit message in patch set 5 08/06/21, 11:55

  8. core-ci voted on patch set 3 V +1 08/06/21, 11:59 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14196

  9. core-ci voted on patch set 4 V +1 08/06/21, 12:03 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14197

  10. Stephan Jorek updated the commit message in patch set 6 08/06/21, 12:05

  11. Stephan Jorek updated the commit message in patch set 7 08/06/21, 12:06

  12. core-ci voted on patch set 5 V +1 08/06/21, 12:09 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14198

  13. Stephan Jorek uploaded patch set 8 08/06/21, 12:14

  14. core-ci voted on patch set 7 V +1 08/06/21, 12:20 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14201

  15. core-ci voted on patch set 8 V +1 08/06/21, 12:31 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14202

  16. Stephan Jorek rebased onto patch set 9 08/06/21, 16:02

  17. core-ci voted on patch set 9 V +1 08/06/21, 16:18 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14217

  18. Benni Mack commented on patch set 9 10/15/21, 09:06

    Hey Stephan, just to understand, would the new function array_is_list() from PHP do the same as your fix? https://www.php.net/manual/en/function.array-is-list.php
  19. Stephan Jorek commented on patch set 9 10/15/21, 09:50

    Hi Benni, indeed this function delivers exactly the same result. Damn cool. Ideally one would use symfony/polyfill-php81 now => https://github.com/symfony/polyfill-php81/blob/main/Php81.php#L21-L36 Shall I refactor the fix accordingly?
  20. Benni Mack commented on patch set 9 10/15/21, 09:57

    Reply

    yeah, ideally we add the polyfill package, and fix the calls TO this function, not the TYPO3 method itself, and document what the TYPO3 method does differently. in v12 we can then deprecate this TYPO3 method (in a separate change)
  21. Stephan Jorek commented on patch set 9 10/15/21, 09:59

    Understand and agree. I'll refactor it just now, to make (y)our life easier …
  22. Stephan Jorek uploaded patch set 10 10/15/21, 10:41

  23. Stephan Jorek uploaded patch set 11 10/15/21, 10:44

  24. core-ci voted on patch set 10 V -1 10/15/21, 10:45 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/18159

  25. core-ci voted on patch set 11 V -1 10/15/21, 10:49 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/18160

  26. Stephan Jorek uploaded patch set 12 10/15/21, 10:49

  27. Stephan Jorek uploaded patch set 13 10/15/21, 10:58

  28. core-ci voted on patch set 12 V +1 10/15/21, 11:01 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/18161

  29. Stephan Jorek uploaded patch set 14 10/15/21, 11:04

  30. Stephan Jorek commented on patch set 14 10/15/21, 11:19

    Reply

    Refactoring has been done: - added symfony/polyfill-php81 to composer.json - use array_is_list in ArrayUtility::isAssociative - use array_is_list in ArrayUtility::replaceAndAppendScalarValuesRecursive - use array_is_list in one other occurrence in the backend-extension's SingleFieldContainer::isAssociativeArray - added another missing unit-test to prove that empty arrays are not associative - updated wording in code-comments - marked ArrayUtility::isAssociative as deprecated with TYPO3 v12 => upps, this was too early, right? Tell me and I'll remove the deprecation mark … - update commit-message accordingly - applied CGL
  31. core-ci voted on patch set 13 V +1 10/15/21, 11:22 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/18164

  32. core-ci voted on patch set 14 V +1 10/15/21, 11:36 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/18166

  33. Benni Mack commented on patch set 14 10/15/21, 12:19

    Let's split up the composer requirement separately https://review.typo3.org/c/Packages/TYPO3.CMS/+/71809
  34. Benni Mack rebased onto patch set 15 10/15/21, 12:50

  35. Oliver Hader rebased onto patch set 16 10/15/21, 19:08

  36. Oliver Hader voted on patch set 16 CR +1 10/15/21, 19:11

  37. core-ci voted on patch set 16 V +1 10/15/21, 19:23 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/18202

  38. Stephan Jorek commented on patch set 16 10/16/21, 08:00

    Thank's Benni and Oliver for the split and the review - was too busy yesterday. Anything left to do?
  39. Helmut Hummel commented on patch set 16 CR -1 10/16/21, 11:53

    Although I very much agree that the current implementation of "ArrayUtility::isAssociative" is wrong and should be fixed, I would suggest to give this one a deeper investigation on the impact of this change, especially because it affects the (generic) YAML parsing and merging, that is currently used in multiple places in TYPO3 and take my comments here https://forge.typo3.org/issues/94729#note-25 into account during such investigation.
  1. Oliver Hader voted on patch set 16 CR -1 10/24/21, 06:30

  2. Benni Mack updated the commit message in patch set 17 11/18/21, 10:42

  3. core-ci voted on patch set 17 V +1 11/18/21, 11:17 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19456

  4. Oliver Hader voted on patch set 17 CR +1 11/18/21, 11:53

  5. Benni Mack wrote 11/29/21, 10:18

    Change destination moved from master to main
  6. Benni Mack updated the commit message in patch set 18 11/29/21, 14:07

  7. core-ci voted on patch set 18 V +1 11/29/21, 18:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20092

  8. Oliver Hader commented on patch set 18 03/21/22, 09:31

    I spotted two other occurrences which seem to be related to checking for a n assoc-array.

    ArrayUtility.php:216

    Seems to be related...

    ArrayUtility.php:846

    Seems to be related...
  9. Benni Mack updated the commit message in patch set 19 V +1 05/03/23, 09:16

  10. core-ci voted on patch set 19 V -1 05/03/23, 09:19 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41388

  11. Oliver Hader rebased onto patch set 20 V -1 05/03/23, 11:54

  12. core-ci voted on patch set 20 V -1 05/03/23, 12:30 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41522

  13. Oliver Hader commented on patch set 21 V -1 05/09/23, 08:08

    Published edit on patch set 20.
  14. Oliver Hader commented on patch set 21 05/09/23, 08:11

    ArrayUtility.php:216 Reply

    Done

    ArrayUtility.php:846 Reply

    Done
  15. core-ci voted on patch set 21 V +1 05/09/23, 08:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41694

  16. Stefan Froemken voted on patch set 21 V +1 CR +1 05/10/23, 19:26

  17. Stefan Bürk commented on patch set 21 05/11/23, 17:40

    Quite unsure. The use of !array_is_liste() for the ArrayUtility::isAssociative() is wrong in my eyes. The native php method approach is quite blue-eyed in my eyes, not really respecing "lived associatve/non-associatve array keys". ```php // returns false, but in my eyes this is a reversed non-associative // array like it is returned by array_reverse() with preserved keys array_is_list([1 => '1', 0 => '0']); // reversed array with preseverd keys $list = ['0', '1', '2']; // should return true, but returns false array_is_list(array_reverse($list, true)); // after an unset of an element, a list should still be a list $list = ['0', '1', '2']; unset($list[1]); // returns false, but it's still not an "associative" array array_is_list($list); ``` and others .... Guessing for the Form Container this may be okay as there should be no reversed order etc. The ArrayUtiliy method is internal, not checked if it is used in other places. If yes, should be checked if really expected bahviour is ensured. Otherwise, a internal marked method must not be deprecated, a simply todo is enough. On the other hand, if marked as deprecated .. we cannot depreacted things in v12 anymore, only in v13 and removed in v14. So better add a todo and remove it in v13 in a dedicated change.
  18. Stephan Jorek commented on patch set 21 05/11/23, 18:01

    Reply

    Right, both cases look like, they should use array_is_list-function instead.
    @stefan bürk What it makes even worse the the following: ```php php > var_dump(array_keys(['10' => 'value'])); php shell code:1: array(1) { [0] => int(10) } ``` Do you see how PHP converts the string-key '10' into an integer key? So this turns this obvious associative array into an array-list. If I combine this behaviour with your (correct) investigation this would result in a bl…y mess. :-/

    Reply

    Wow - a very good catch! The only thing, which makes me a little bit sad, is that I raised this issue 2 years ago and still it's neither solved in v10 nor in v11, which introduced this glitch in yaml-imports. But seriously, I have to admit that you're right … to my mind this should be solved directly in PHP (and in symfony's php81 shim). Anybody willing to raise an issue on https://bugs.php.net/? I already have an idea, how to catch this in PHP's C implementation, but no good idea how to do this in plain PHP (for symfony). If I get no reply, I'll do it this weekend with special kudos to you.
  19. Stefan Bürk commented on patch set 21 05/11/23, 18:05

    Reply

    Did not checked deeper, was just my first thoughts by only reading the change/code. Therefore, no vote yet in any direction.
  20. Stefan Bürk commented on patch set 21 05/11/23, 18:11

    Reply

    Sure. The point is that for php you could use numbered index as string and php will handle them like integer one. Nativly spoken, php array's are always associatve per nature. And `array_is_list()` not really helps here. And integer like string array keys are handled as integers in some circumstances, which makes this a mess. The symfony polyfill provides a polyfill which works exactly like the native `array_is_list()` method, so from that point it's okay. The messy point is, that `array_is_list()` checks for only one concrete list implementation with the consecutive UP-wards counter for keys. Which per nature are not all kind of lists, so a relativly naive approch. At least in my eyes. But as said, it may be okay in case for the "yaml import" handling - but not checked yet.
  21. Stefan Froemken wrote 05/22/23, 07:54

    Removed reviewer Stefan Froemken with the following votes: * Verified+1 by Stefan Froemken * Code-Review+1 by Stefan Froemken
  22. Benni Mack updated the commit message in patch set 22 V +1 10/18/24, 14:07

  23. Benni Mack commented on patch set 22 04/11/25, 21:23

    I dont understand this change tbh... Reading through the tickets and the problems there this change does not fix a bug in EXT:form?. Also: we do not use isAssociative and we can remove it. But what to achieve then?
  24. Stefan Bürk commented on patch set 22 04/12/25, 13:15

    Reply

    Done
  25. Stefan Bürk abandoned the change 04/12/25, 13:17

    The main goal of this change (ArrayUtility + ext:forms) has been resolved meanwhile with https://review.typo3.org/c/Packages/TYPO3.CMS/+/73984 using the same solution preferred here first, which vanished due to the rebases. The deprecation of the ArrayUtility method is not what we strives for. The backend form-engine adjustment was a side change and will be done in a dedicated change, as it is unrelated to the issue and change here.