Review main Merged

85856[BUGFIX] Resolve references of extension forms

TYPO3com CR +2 V +2 Hard +352 −12 updated 11/18/24, 14:27

Commit message

The EXT:form management GUI shows all forms and where
they are used (referenced).

Forms can be either stored in FAL-based storages
(mainly used by editors) or be shipped in extensions
(EXT:my_extension/Resources/Private/Forms/my.form.yaml).

Clicking on the number of references opens a popup
that reveals all references to a given form, but this
could not deal with EXT:... notations and throwing
an exception, instead of returning the sys_refindex
records that actually contain the exact reference.

This patch now resolves the sys_refindex differently,
depending on whether these notations are used:

* FAL-based storage with UID
* FAL-based storage with string
* public EXT:... notation
* private EXT:... notation

Tests are added to ensure this behavior.

Inspecting references will no longer trigger an
exception, but an empty result set to continue
the sys_refindex inspection.

Review

  • Code review CR +2 Done
    • Garvin Hicking +2
    • Christian Kuhn +1
    • Mathias Brodala +1
    • André Buchmann +1
  • Verified V +2 Done
    • Garvin Hicking +2
    • Christian Kuhn +1
    • André Buchmann +1
    • core-ci +1 CI

Issues 1

Files 11

Review log 3850

Show 1625 earlier messages
  1. TYPO3com uploaded patch set 1 09/03/24, 14:40

  2. core-ci voted on patch set 1 V -1 09/03/24, 14:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64672

  3. core-ci commented on patch set 1 09/03/24, 16:05 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64672

  4. TYPO3com rebased onto patch set 2 V -1 09/05/24, 07:50

  5. core-ci voted on patch set 2 V -1 09/05/24, 07:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64839

  6. Markus Zipfel uploaded patch set 3 V -1 09/05/24, 08:26

  7. core-ci voted on patch set 3 V +1 09/05/24, 08:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64846

  8. Achim Fritz voted on patch set 3 V +1 09/16/24, 09:33

  9. Christian Kuhn commented on patch set 3 09/17/24, 13:47

    Half confirmed the issue: It seems, his only happens with *composer instances*, so not reproducible with a plain monorepo instance. I'm not 100% sure about the solution currently, it needs a bit more investigation. I actually hate the fact that ext:form misuses sys_refindex, needs a custom hook to do this, plus a very ugly index on softref_key in DB. I'm very unhappy with the situation that sys_refindex carries these references to FAL *and* EXT:... files overall as softref overall, and I think this needs to vanish somehow. But the main point is like this: The two-folded file stuff in ext:form (having forms in FAL plus having them in EXT:...) leads to a plethora of issues and hacks throughout ext:form, including security related code to avoid editors without permissions fiddling with such files in file-list module and all that. With my latest ext:form refactoring, I came to the conclusion we have to establish a new solution for that within ext:form that dissolves the situation, most likely by always storing forms at a third place out of extension scope (with some sync mechanic) and FAL scope. This is however much more work and needs some dedicated effort. We should do it, but it's no quick win. For this patch: I'm open to *somehow* fix this reference thing for now. This patch should have a good review and testing, and probably some fine tuning applied.
  10. Markus Zipfel commented on patch set 3 09/17/24, 16:27

    Reply

    100% agree on that! I currently can not think of an easy way to bring together the possibility of (deployable) form YAML files from extension resources **and** regular form YAML FAL files from fileadmin. Also I just checked the issue in a fresh 13.3 environment and I do not have the problem there. I will try to reproduce the error tomorrow.
  11. Markus Zipfel commented on patch set 3 09/27/24, 08:16

    Reply

    > I will try to reproduce the error tomorrow. Sorry, I did not find the time until now. Just checked it with a fresh 13.3.0 system in composer mode: The error still exists (and the patch solves it). > I'm open to _somehow_ fix this reference thing for now. This patch should have a good review and testing, and probably some fine tuning applied. @lolli@schwarzbu.ch If I can do some refactoring or other (simple) changes I would be glad to help.
  12. Mathias Brodala updated the commit message in patch set 4 V +1 09/27/24, 10:19

  13. Mathias Brodala voted on patch set 4 V +1 09/27/24, 10:19

  14. Mathias Brodala commented on patch set 4 CR -1 09/27/24, 10:23

    DatabaseService.php:51

    Since we know this must be a file (no matter if it exists), also looking up folders should be avoided. Thus $resourceFactory->getFileObject() should be used instead. Then of course taking FileDoesNotExistException into account.

    DatabaseService.php:55

    This should be moved upwards to be an early return instead.

    DatabaseService.php:59

    Maybe do a simple check for the "EXT:" prefix early instead?
  15. core-ci voted on patch set 4 V +1 09/27/24, 10:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66733

  16. Oliver Hader updated the commit message in patch set 5 CR -1 V +1 10/18/24, 14:02

  17. Mathias Brodala commented on patch set 5 CR -1 10/25/24, 08:06

    See previous review
  18. Mathias Brodala voted on patch set 5 V +1 10/25/24, 08:06

  19. Garvin Hicking commented on patch set 5 10/25/24, 08:11

    on it.
  20. Benjamin Franzke commented on patch set 5 CR -1 10/25/24, 08:18

    This must not use a fail-first strategy, it instead needs the same approach as the following change: >71512: [BUGFIX] Move virtual storage 0 back to public path | https://review.typo3.org/c/Packages/TYPO3.CMS/+/71512 Check for extension path via `PathUtility::isExtensionPath` before even trying to use FAL at all.
  21. Garvin Hicking uploaded patch set 6 CR -1 V +1 10/25/24, 23:20

  22. core-ci voted on patch set 6 V -1 10/25/24, 23:28 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68645

  23. Garvin Hicking uploaded patch set 7 V -1 10/25/24, 23:37

    Fix mariadb oddity for space identical to empty condition
  24. core-ci voted on patch set 7 V +1 10/25/24, 23:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68646

  25. Garvin Hicking commented on patch set 7 V +1 10/25/24, 23:46

    I reworked this and added tests and believe this is a good interim solution to make refindexes work again. In the end this patch/implementation just tries to gracefully fail on reading the sys_refindex for yielding records. Further access restrictions on the records would be evaluated downstream or in the follow-up when a user actually tries to edit a referenced index. I hope this maybe can move things forward, and having the tests would allow for future refactoring and securing functionality. (I agree that the mixing of FAL and non-FAL creates conceptual trouble. But having this option is also a huge benefit in terms of deployable content, and locking away editing of specific forms from editors. Figuring out a "proper" solution to this will surely cost a hair, or too.)
  1. Garvin Hicking commented on patch set 7 10/25/24, 23:49

    DatabaseService.php:51 Reply

    Actually getFileObject can't deal with strings and this method also is kind of advertised as the god-method "Bulk function, can be used for anything to get a file or folder". I believe this is a good fit for the current catch-all scenario, hope it's okay to resolve this one.

    DatabaseService.php:55 Reply

    Reworked (heavily), with a focus on readability and not "minimal amount of code-forks". I hope that has a benefit, but am open for suggestions on how to refactor. Resolving for now.

    DatabaseService.php:59 Reply

    Done
  2. Mathias Brodala commented on patch set 7 10/28/24, 07:39

    DatabaseService.php:63

    $useStringReference would be clearer IMO. Both queries are valid, querying "ref_string" is not really a fallback in case "ref_uid" is not queried.
  3. Garvin Hicking commented on patch set 8 V +1 10/28/24, 08:13

    Published edit on patch set 7.
  4. Garvin Hicking commented on patch set 8 10/28/24, 08:13

    DatabaseService.php:63 Reply

    Good idea, yes. Changed accordingly.
  5. core-ci voted on patch set 8 V +1 10/28/24, 08:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68711

  6. Mathias Brodala commented on patch set 8 CR -1 10/28/24, 08:44

    DatabaseServiceTest.php:149

    From reading this should actually be 3 separate tests: - getReferencesByPersistenceIdentifierAcceptsValidFile(string $yamlFile) - getReferencesByPersistenceIdentifierRejectsInvalidIdentifier(string $yamlFile) - getReferencesByPersistenceIdentifierSkipsMissingFile(string $yamlFile) This would make clearer what each test is about and simplify the data provider a lot. Also it won't get worse if even more conditions are added in the future. Tests themselves should not contain any conditional code, otherwise you actually need tests for the tests.

    DatabaseServiceTest.php:164

    Better not have a silent fallback here. This should fail with an "undefined index" error to actually raise attention.
  7. Garvin Hicking commented on patch set 8 10/28/24, 08:52

    DatabaseServiceTest.php:149 Reply

    I understand. I won't have time to rework the tests in the next week. Also will wait for feedback from @lolli@schwarzbu.ch before I put more time into it. thanks!
  8. Garvin Hicking uploaded patch set 9 CR -1 V +1 10/30/24, 10:09

    Adjust test
  9. core-ci voted on patch set 9 V +1 10/30/24, 10:14 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68838

  10. Garvin Hicking commented on patch set 9 10/30/24, 10:49

    DatabaseServiceTest.php:149 Reply

    This was very helpful and not much work actually, now that I properly understood the task. Thanks for pointing me at it. That makes this test MUCH better from my POV. Please see if that works for you. :-)

    DatabaseServiceTest.php:164 Reply

    Done
  11. Mathias Brodala commented on patch set 9 CR +1 10/30/24, 11:14

    DatabaseServiceTest.php:149 Reply

    Great, this is a lot easier to understand!
  12. Benjamin Franzke commented on patch set 9 11/11/24, 13:45

    Reply

    @gh@faktor-e.de This aspect (parsing EXT-notation-first, before trying to hand it into FAL) looks good from a security and consistency (with the other resolution algorithms in EXT:form) perspective now. Thanks! I can not comment on the refindex lookups in detail for now. That would be better suited for @lolli@schwarzbu.ch
  13. Christian Kuhn commented on patch set 9 11/11/24, 13:53

    will try to catch up with this soon.
  14. André Buchmann commented on patch set 9 V +1 11/14/24, 09:35

    Can confirm this patch works with 12.4.23 (applied with composer patches)
  15. Christian Kuhn commented on patch set 9 11/18/24, 12:54

    Reply

    i think we'll go with the patch as advertised for now. we'll still have to rethink the overall situation at some point, but that can of course only be a thing for later.
  16. Christian Kuhn uploaded patch set 10 CR +1 V +1 11/18/24, 13:23

  17. core-ci voted on patch set 10 V +1 11/18/24, 13:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69594

  18. Mathias Brodala voted on patch set 10 CR +1 11/18/24, 13:39

  19. André Buchmann voted on patch set 10 CR +1 V +1 11/18/24, 13:42

  20. Garvin Hicking voted on patch set 10 CR +1 V +1 11/18/24, 13:50

  21. Christian Kuhn voted on patch set 10 CR +1 V +1 11/18/24, 14:20

  22. Christian Kuhn commented on patch set 10 11/18/24, 14:21

    great to see this solved. thanks everyone for work on this!
  23. Garvin Hicking voted on patch set 10 CR +2 11/18/24, 14:25

  24. Garvin Hicking voted on patch set 10 V +2 11/18/24, 14:25

  25. Garvin Hicking merged patch set 11 11/18/24, 14:27