85856[BUGFIX] Resolve references of extension forms
Commit message
Review
-
Code review CR +2 Done
- Garvin Hicking +2
- Christian Kuhn +1
- Mathias Brodala +1
- André Buchmann +1
-
Verified V +2 Done
- Garvin Hicking +2
- Christian Kuhn +1
- André Buchmann +1
- core-ci +1 CI
Issues 1
Files 11
- typo3/sysext/form/Classes/Service/DatabaseService.php8+69 −12
- typo3/sysext/form/Tests/Functional/Service/Fixtures/Extensions/form_references/Resources/Private/Forms/contact1.form.yamlnew+3 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/Extensions/form_references/Resources/Public/Forms/contact2.form.yamlnew+3 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/Extensions/form_references/composer.jsonnew+14 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/Extensions/form_references/ext_emconf.phpnew+21 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/FileadminOffline/Forms/contact5.form.yamlnew+3 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/FileadminPrivate/Forms/contact3.form.yamlnew+3 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/FileadminPublic/Forms/contact4.form.yamlnew+3 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/FileadminPublic/Forms/contact6.form.yamlnew+3 −0
- typo3/sysext/form/Tests/Functional/Service/Fixtures/DatabaseServiceTestImport.csvnew+76 −0
- typo3/sysext/form/Tests/Functional/Service/DatabaseServiceTest.php6new+154 −0
Review log 3850
Show 1625 earlier messages
-
TYPO3com uploaded patch set 1 09/03/24, 14:40
-
core-ci voted on patch set 1 V -1 09/03/24, 14:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64672
-
core-ci commented on patch set 1 09/03/24, 16:05 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64672
-
TYPO3com rebased onto patch set 2
V -109/05/24, 07:50 -
core-ci voted on patch set 2 V -1 09/05/24, 07:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64839
-
Markus Zipfel uploaded patch set 3
V -109/05/24, 08:26 -
core-ci voted on patch set 3 V +1 09/05/24, 08:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/64846
-
Achim Fritz voted on patch set 3 V +1 09/16/24, 09:33
-
Christian Kuhn commented on patch set 3 09/17/24, 13:47
Half confirmed the issue: It seems, his only happens with *composer instances*, so not reproducible with a plain monorepo instance. I'm not 100% sure about the solution currently, it needs a bit more investigation. I actually hate the fact that ext:form misuses sys_refindex, needs a custom hook to do this, plus a very ugly index on softref_key in DB. I'm very unhappy with the situation that sys_refindex carries these references to FAL *and* EXT:... files overall as softref overall, and I think this needs to vanish somehow. But the main point is like this: The two-folded file stuff in ext:form (having forms in FAL plus having them in EXT:...) leads to a plethora of issues and hacks throughout ext:form, including security related code to avoid editors without permissions fiddling with such files in file-list module and all that. With my latest ext:form refactoring, I came to the conclusion we have to establish a new solution for that within ext:form that dissolves the situation, most likely by always storing forms at a third place out of extension scope (with some sync mechanic) and FAL scope. This is however much more work and needs some dedicated effort. We should do it, but it's no quick win. For this patch: I'm open to *somehow* fix this reference thing for now. This patch should have a good review and testing, and probably some fine tuning applied. -
Markus Zipfel commented on patch set 3 09/17/24, 16:27
Reply
100% agree on that! I currently can not think of an easy way to bring together the possibility of (deployable) form YAML files from extension resources **and** regular form YAML FAL files from fileadmin. Also I just checked the issue in a fresh 13.3 environment and I do not have the problem there. I will try to reproduce the error tomorrow. -
Markus Zipfel commented on patch set 3 09/27/24, 08:16
Reply
> I will try to reproduce the error tomorrow. Sorry, I did not find the time until now. Just checked it with a fresh 13.3.0 system in composer mode: The error still exists (and the patch solves it). > I'm open to _somehow_ fix this reference thing for now. This patch should have a good review and testing, and probably some fine tuning applied. @lolli@schwarzbu.ch If I can do some refactoring or other (simple) changes I would be glad to help. -
Mathias Brodala updated the commit message in patch set 4
V +109/27/24, 10:19 -
Mathias Brodala voted on patch set 4 V +1 09/27/24, 10:19
-
Mathias Brodala commented on patch set 4 CR -1 09/27/24, 10:23
DatabaseService.php:51Since we know this must be a file (no matter if it exists), also looking up folders should be avoided. Thus $resourceFactory->getFileObject() should be used instead. Then of course taking FileDoesNotExistException into account.DatabaseService.php:55This should be moved upwards to be an early return instead.DatabaseService.php:59Maybe do a simple check for the "EXT:" prefix early instead? -
core-ci voted on patch set 4 V +1 09/27/24, 10:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66733
-
Oliver Hader updated the commit message in patch set 5
CR -1V +110/18/24, 14:02 -
Mathias Brodala commented on patch set 5 CR -1 10/25/24, 08:06
See previous review -
Mathias Brodala voted on patch set 5 V +1 10/25/24, 08:06
-
Garvin Hicking commented on patch set 5 10/25/24, 08:11
on it. -
Benjamin Franzke commented on patch set 5 CR -1 10/25/24, 08:18
This must not use a fail-first strategy, it instead needs the same approach as the following change: >71512: [BUGFIX] Move virtual storage 0 back to public path | https://review.typo3.org/c/Packages/TYPO3.CMS/+/71512 Check for extension path via `PathUtility::isExtensionPath` before even trying to use FAL at all. -
Garvin Hicking uploaded patch set 6
CR -1V +110/25/24, 23:20 -
core-ci voted on patch set 6 V -1 10/25/24, 23:28 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68645
-
Garvin Hicking uploaded patch set 7
V -110/25/24, 23:37Fix mariadb oddity for space identical to empty condition -
core-ci voted on patch set 7 V +1 10/25/24, 23:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68646
-
Garvin Hicking commented on patch set 7 V +1 10/25/24, 23:46
I reworked this and added tests and believe this is a good interim solution to make refindexes work again. In the end this patch/implementation just tries to gracefully fail on reading the sys_refindex for yielding records. Further access restrictions on the records would be evaluated downstream or in the follow-up when a user actually tries to edit a referenced index. I hope this maybe can move things forward, and having the tests would allow for future refactoring and securing functionality. (I agree that the mixing of FAL and non-FAL creates conceptual trouble. But having this option is also a huge benefit in terms of deployable content, and locking away editing of specific forms from editors. Figuring out a "proper" solution to this will surely cost a hair, or too.)
-
Garvin Hicking commented on patch set 7 10/25/24, 23:49
DatabaseService.php:51ReplyActually getFileObject can't deal with strings and this method also is kind of advertised as the god-method "Bulk function, can be used for anything to get a file or folder". I believe this is a good fit for the current catch-all scenario, hope it's okay to resolve this one.DatabaseService.php:55ReplyReworked (heavily), with a focus on readability and not "minimal amount of code-forks". I hope that has a benefit, but am open for suggestions on how to refactor. Resolving for now.DatabaseService.php:59ReplyDone -
Mathias Brodala commented on patch set 7 10/28/24, 07:39
DatabaseService.php:63$useStringReference would be clearer IMO. Both queries are valid, querying "ref_string" is not really a fallback in case "ref_uid" is not queried. -
Garvin Hicking commented on patch set 8
V +110/28/24, 08:13Published edit on patch set 7. -
Garvin Hicking commented on patch set 8 10/28/24, 08:13
DatabaseService.php:63ReplyGood idea, yes. Changed accordingly. -
core-ci voted on patch set 8 V +1 10/28/24, 08:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68711
-
Mathias Brodala commented on patch set 8 CR -1 10/28/24, 08:44
DatabaseServiceTest.php:149From reading this should actually be 3 separate tests: - getReferencesByPersistenceIdentifierAcceptsValidFile(string $yamlFile) - getReferencesByPersistenceIdentifierRejectsInvalidIdentifier(string $yamlFile) - getReferencesByPersistenceIdentifierSkipsMissingFile(string $yamlFile) This would make clearer what each test is about and simplify the data provider a lot. Also it won't get worse if even more conditions are added in the future. Tests themselves should not contain any conditional code, otherwise you actually need tests for the tests.DatabaseServiceTest.php:164Better not have a silent fallback here. This should fail with an "undefined index" error to actually raise attention. -
Garvin Hicking commented on patch set 8 10/28/24, 08:52
DatabaseServiceTest.php:149ReplyI understand. I won't have time to rework the tests in the next week. Also will wait for feedback from @lolli@schwarzbu.ch before I put more time into it. thanks! -
Garvin Hicking uploaded patch set 9
CR -1V +110/30/24, 10:09Adjust test -
core-ci voted on patch set 9 V +1 10/30/24, 10:14 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68838
-
Garvin Hicking commented on patch set 9 10/30/24, 10:49
DatabaseServiceTest.php:149ReplyThis was very helpful and not much work actually, now that I properly understood the task. Thanks for pointing me at it. That makes this test MUCH better from my POV. Please see if that works for you. :-)DatabaseServiceTest.php:164ReplyDone -
Mathias Brodala commented on patch set 9 CR +1 10/30/24, 11:14
DatabaseServiceTest.php:149ReplyGreat, this is a lot easier to understand! -
Benjamin Franzke commented on patch set 9 11/11/24, 13:45
Reply
@gh@faktor-e.de This aspect (parsing EXT-notation-first, before trying to hand it into FAL) looks good from a security and consistency (with the other resolution algorithms in EXT:form) perspective now. Thanks! I can not comment on the refindex lookups in detail for now. That would be better suited for @lolli@schwarzbu.ch -
Christian Kuhn commented on patch set 9 11/11/24, 13:53
will try to catch up with this soon. -
André Buchmann commented on patch set 9 V +1 11/14/24, 09:35
Can confirm this patch works with 12.4.23 (applied with composer patches) -
Christian Kuhn commented on patch set 9 11/18/24, 12:54
Reply
i think we'll go with the patch as advertised for now. we'll still have to rethink the overall situation at some point, but that can of course only be a thing for later. -
Christian Kuhn uploaded patch set 10
CR +1V +111/18/24, 13:23 -
core-ci voted on patch set 10 V +1 11/18/24, 13:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69594
-
Mathias Brodala voted on patch set 10 CR +1 11/18/24, 13:39
-
André Buchmann voted on patch set 10 CR +1 V +1 11/18/24, 13:42
-
Garvin Hicking voted on patch set 10 CR +1 V +1 11/18/24, 13:50
-
Christian Kuhn voted on patch set 10 CR +1 V +1 11/18/24, 14:20
-
Christian Kuhn commented on patch set 10 11/18/24, 14:21
great to see this solved. thanks everyone for work on this! -
Garvin Hicking voted on patch set 10 CR +2 11/18/24, 14:25
-
Garvin Hicking voted on patch set 10 V +2 11/18/24, 14:25
-
Garvin Hicking merged patch set 11 11/18/24, 14:27