70814[BUGFIX] Support parsing of XML files larger than 10 MB
Commit message
Review
-
Code review CR +2 Done
- Andreas Kienast +2
- Benni Mack +1
-
Verified V +2 Done
- Andreas Kienast +2
- Benni Mack +1
- core-ci +1 CI
Issues 1
Files 11
- Build/phpstan/phpstan-baseline.neon+5 −0
- typo3/sysext/core/Classes/Serializer/Exception/InvalidDataException.phpnew+27 −0
- typo3/sysext/core/Classes/Serializer/Typo3XmlParser.phpnew+250 −0
- typo3/sysext/core/Classes/Serializer/Typo3XmlParserOptions.phpnew+71 −0
- typo3/sysext/core/Classes/Serializer/Typo3XmlSerializer.phpnew+340 −0
- typo3/sysext/core/Classes/Serializer/Typo3XmlSerializerOptions.phpnew+79 −0
- typo3/sysext/core/Tests/Unit/Serializer/Fixtures/file.gifnewbinary
- typo3/sysext/core/Tests/Unit/Serializer/Typo3XmlParserTest.phpnew+659 −0
- typo3/sysext/core/Tests/Unit/Serializer/Typo3XmlSerializerTest.phpnew+522 −0
- typo3/sysext/impexp/Classes/Export.php+7 −1
- typo3/sysext/impexp/Classes/Import.php+47 −22
Review log 114177
Show 96152 earlier messages
-
Alexander Nitsche uploaded patch set 1 08/29/21, 16:06
-
core-ci voted on patch set 1 V -1 08/29/21, 16:09 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15824
-
Alexander Nitsche uploaded patch set 2 08/30/21, 15:46
-
core-ci voted on patch set 2 V -1 08/30/21, 15:51 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15881
-
Alexander Nitsche uploaded patch set 3 08/30/21, 15:57
-
core-ci voted on patch set 3 V -1 08/30/21, 16:18 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15882
-
Alexander Nitsche uploaded patch set 4 08/30/21, 16:36
-
core-ci voted on patch set 4 V -1 08/30/21, 16:50 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15884
-
Benni Mack commented on patch set 4 08/31/21, 19:30
Please do not put this in 9.5 (or even 10.4) -
Alexander Nitsche uploaded patch set 5 09/01/21, 07:37
-
Alexander Nitsche commented on patch set 5 09/01/21, 07:39
Reply
Ok. -
core-ci voted on patch set 5 V -1 09/01/21, 07:43 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15984
-
Alexander Nitsche uploaded patch set 6 09/01/21, 17:26
-
core-ci voted on patch set 6 V +1 09/01/21, 17:50 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16021
-
Alexander Nitsche uploaded patch set 7 09/02/21, 11:48
-
core-ci voted on patch set 7 V -1 09/02/21, 12:11 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16071
-
Alexander Nitsche uploaded patch set 8 09/02/21, 15:28
-
core-ci voted on patch set 8 V -1 09/02/21, 15:35 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16107
-
Alexander Nitsche uploaded patch set 9 09/03/21, 15:49
-
core-ci voted on patch set 9 V -1 09/03/21, 15:53 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16204
-
Alexander Nitsche uploaded patch set 10 09/04/21, 08:29
-
core-ci voted on patch set 10 V -1 09/04/21, 08:48 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16232
-
Alexander Nitsche uploaded patch set 11 09/04/21, 13:04
-
core-ci voted on patch set 11 V -1 09/04/21, 13:26 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16245
-
Alexander Nitsche uploaded patch set 12 09/04/21, 16:59
-
core-ci voted on patch set 12 V -1 09/04/21, 17:27 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16269
-
Alexander Nitsche uploaded patch set 13 09/04/21, 19:26
-
core-ci voted on patch set 13 V -1 09/04/21, 19:47 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16271
-
Alexander Nitsche uploaded patch set 14 09/05/21, 09:53
-
core-ci voted on patch set 14 V -1 09/05/21, 10:06 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16282
-
Alexander Nitsche uploaded patch set 15 09/06/21, 08:51
-
core-ci voted on patch set 15 V -1 09/06/21, 09:07 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16311
-
Alexander Nitsche uploaded patch set 16 09/06/21, 15:57
-
core-ci voted on patch set 16 V -1 09/06/21, 16:13 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16336
-
Alexander Nitsche uploaded patch set 17 09/07/21, 13:45
-
core-ci voted on patch set 17 V -1 09/07/21, 14:05 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16377
-
Alexander Nitsche uploaded patch set 18 09/07/21, 14:49
-
core-ci voted on patch set 18 V -1 09/07/21, 14:53 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16379
-
Alexander Nitsche uploaded patch set 19 09/08/21, 17:13
-
core-ci voted on patch set 19 V -1 09/08/21, 17:35 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16423
-
Alexander Nitsche uploaded patch set 20 09/09/21, 12:01
-
core-ci voted on patch set 20 V -1 09/09/21, 12:06 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16448
-
Alexander Nitsche uploaded patch set 21 09/09/21, 12:10
-
core-ci voted on patch set 21 V -1 09/09/21, 14:14 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16449
-
Alexander Nitsche uploaded patch set 22 09/09/21, 14:47
-
Alexander Nitsche uploaded patch set 23 09/09/21, 16:04
-
core-ci voted on patch set 22 V -1 09/09/21, 16:51 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16460
-
core-ci voted on patch set 23 V -1 09/09/21, 18:08 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16464
-
Alexander Nitsche uploaded patch set 24 09/10/21, 12:19
-
core-ci voted on patch set 24 V -1 09/10/21, 14:30 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16479
-
Alexander Nitsche uploaded patch set 25 09/10/21, 17:01
-
Benjamin Franzke uploaded patch set 26 09/10/21, 18:34
-
core-ci voted on patch set 25 V -1 09/10/21, 19:04 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16505
-
core-ci voted on patch set 26 V -1 09/10/21, 20:38 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16510
-
Alexander Nitsche uploaded patch set 27 09/13/21, 14:37
-
core-ci voted on patch set 27 V -1 09/13/21, 14:54 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16659
-
Alexander Nitsche uploaded patch set 28 09/13/21, 16:52
-
core-ci voted on patch set 28 V -1 09/13/21, 17:10 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16666
-
Alexander Nitsche uploaded patch set 29 09/13/21, 17:22
-
core-ci voted on patch set 29 V -1 09/13/21, 17:37 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16671
-
Alexander Nitsche uploaded patch set 30 09/13/21, 20:19
-
core-ci voted on patch set 30 V -1 09/13/21, 20:33 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16677
-
Alexander Nitsche uploaded patch set 31 09/13/21, 21:41
-
core-ci voted on patch set 31 V -1 09/13/21, 21:52 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16680
-
Alexander Nitsche uploaded patch set 32 09/14/21, 09:03
-
core-ci voted on patch set 32 V -1 09/14/21, 09:18 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16690
-
Alexander Nitsche uploaded patch set 33 09/14/21, 14:13
-
core-ci voted on patch set 33 V -1 09/14/21, 14:26 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16708
-
Alexander Nitsche uploaded patch set 34 09/14/21, 14:33
-
core-ci voted on patch set 34 V -1 09/14/21, 14:49 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16710
-
Alexander Nitsche updated the commit message in patch set 35 09/14/21, 15:31
-
core-ci voted on patch set 35 V -1 09/14/21, 15:44 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16713
-
Alexander Nitsche commented on patch set 35 09/17/21, 15:05
This patch is almost done: there are only PHPstan errors left. i would highly appreciate some experienced look at - typo3/sysext/core/Classes/DataHandling/DataHandler.php which requires to fix 3 phpstan issues where the problem is that the used method did not strictly specify the return type before and now, as it does, phpstan does not like it, and i do not feel comfortable in that area to make changes there. -
Alexander Nitsche updated the commit message in patch set 36 09/24/21, 08:02
-
core-ci voted on patch set 36 V -1 09/24/21, 08:31 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17137
-
Andreas Kienast commented on patch set 36 CR -1 09/24/21, 08:35
on it -
Andreas Kienast uploaded patch set 37 09/24/21, 08:56
-
Andreas Kienast commented on patch set 37 09/24/21, 08:56
Rebased only, phpstan issues remain -
Andreas Kienast uploaded patch set 38 09/24/21, 09:04
-
core-ci voted on patch set 37 V -1 09/24/21, 09:07 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17142
-
Andreas Kienast uploaded patch set 39 09/24/21, 09:10
-
core-ci voted on patch set 38 V +1 09/24/21, 09:17 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17143
-
core-ci voted on patch set 39 V +1 09/24/21, 09:29 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17145
-
Andreas Kienast voted on patch set 39 V +1 CR +1 09/24/21, 10:19
-
Oliver Hader commented on patch set 39 CR -1 09/24/21, 10:49
I did not have time to fully test this change, but discovered a bunch of hacks that should be avoided. I'm wondering if there's no alternative on packagist.org. Besides that I'd prefer a stream-based reader, and using visitor patterns... -
Oliver Hader commented on patch set 39 09/24/21, 12:08
Okay... I had a closer look now... * basically the fact that a lot of classes are touched and changed puzzled me * I guess it would be possible to use new `XmlEncoder` (or a similar name) in existing `GeneralUtility::xml2array` and similar functions, wouldn't it? * Some XML-spec related flaws (that exist in current `GeneralUtility` implementation) still could be tackled * Forget my remark about a vistor-based parser, since current consumers expect a full blown array already... -
Alexander Nitsche commented on patch set 39 09/24/21, 12:50
Reply
The switch from the GeneralUtility XML functions to an own class dedicated to XML was to encapsulate general XML handling, make it testable without having to mock caching, making it injectible and having the full set of public and protected functions for encapsulating functionality at hand without having it all static. All occurrences of the old functions have been replaced by the new once in this patch, this is why there are so many files touched, yes. But this will not have to be done soon again, even if the implementation details change in XMLEncoder in the future. I would be eager to tackle XML-spec related flaws. Maybe in a follow-up patch, but nevertheless .. which come to your mind? Btw: One thing left is the former caching layer of xml-to-array decoding which has been removed in this implementation. Benni and Lolli wanted to check, if it is still required. -
Alexander Nitsche uploaded patch set 40 09/29/21, 09:06
-
core-ci voted on patch set 40 V +1 09/29/21, 09:24 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17305
-
Alexander Nitsche commented on patch set 40 09/29/21, 09:25
> Benni: > I would suggest you make it two patches that build on each other: > - New class + tests + adjustments to EXT:impexp > - Adjustments in the other places everywhere Done. Split into this patch #83580 (EXT:impexp only) and new patch #95398 (other places). -
Oliver Hader commented on patch set 40 10/04/21, 08:55
This change looks better now, since it does not roll out the usage to "all" core classes. If just have one remark concerning @internal in XmlEncoder.XmlEncoder.php:27Can we make this class `@internal` for the time being? I see more potential in adjusting the "configuration" and make that more explicit using other techniques (e.g. using a builder pattern instead of the current array-based configuration approach). -
Alexander Nitsche uploaded patch set 41 10/04/21, 08:56
-
core-ci voted on patch set 41 V +1 10/04/21, 09:10 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17537
-
Benni Mack uploaded patch set 42 10/25/21, 17:22
-
core-ci voted on patch set 42 V -1 10/25/21, 17:56 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18549
-
Benni Mack uploaded patch set 43 10/25/21, 18:19
-
core-ci voted on patch set 43 V -1 10/25/21, 18:34 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18552
-
Benni Mack uploaded patch set 44 11/03/21, 16:48
-
core-ci voted on patch set 44 V -1 11/03/21, 17:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18800
-
Benni Mack wrote 11/29/21, 10:15
Change destination moved from master to main -
Benni Mack updated the commit message in patch set 45 11/29/21, 14:03
-
core-ci voted on patch set 45 V -1 11/29/21, 18:27 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20082
-
Alexander Nitsche uploaded patch set 46 01/12/22, 13:25
-
core-ci voted on patch set 46 V +1 01/12/22, 13:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/21971
-
Alexander Nitsche updated the commit message in patch set 47 01/20/22, 16:02
-
core-ci voted on patch set 47 V +1 01/20/22, 16:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22309
-
Alexander Nitsche uploaded patch set 48 01/20/22, 17:36
-
core-ci voted on patch set 48 V +1 01/20/22, 17:53 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22314
-
Alexander Nitsche commented on patch set 48 01/21/22, 12:49
Any dependency injection instantiation of XmlEncoder and XmlDecoder will be postponed to the follow-up patch https://forge.typo3.org/issues/95398 which rolls out the use of both classes to the complete TYPO3 Core. -
Andreas Kienast commented on patch set 48 CR -1 01/21/22, 14:12
XmlDecoder.php:49): array|stringXmlDecoder.php:64): array|stringXmlDecoder.php:81Can be removed, we require PHP 8.1 nowXmlDecoder.php:164): array|string|nullXmlDecoder.php:171[EA] Third parameter should be provided to clarify if type safety is important in this context.XmlDecoder.php:193[EA] '$key[0]' might be used instead (invalid index accesses might show up).XmlDecoder.php:201Something is fishy here, PhpStorm reports the following: Expected parameter of type 'string', 'array' providedXmlDecodingOptions.php:46This allows passing unknown keys without the possibility to fetch them again. Should we filter them before merging with $this->options? Also, array_merge() would remove any "default" options, if I override the key, this should be documented IMO. ``` php > var_dump(array_merge(['a' => ['b', 'c']], ['a' => ['e', 'f']])); php shell code:1: array(1) { 'a' => array(2) { [0] => string(1) "e" [1] => string(1) "f" } } ```XmlEncoder.php:139...): string {XmlEncoder.php:274Is the loose comparison intended?XmlEncoder.php:323mixed $valueXmlEncoder.php:336cast is superfluous as it's already checked aboveXmlEncoder.php:338loose comparisonXmlEncodingOptions.php:67Can be converted to ``` return match ($this->options[self::FORMAT]) { self::FORMAT_INLINE => '', self::FORMAT_PRETTY_WITH_TAB => "\t", default => str_repeat(' ', max(0, $this->options[self::FORMAT])), }; ``` -
Alexander Nitsche uploaded patch set 49 01/21/22, 16:01
-
Alexander Nitsche commented on patch set 49 01/21/22, 16:11
XmlDecoder.php:49ReplyDoneXmlDecoder.php:64ReplyDoneXmlDecoder.php:81ReplyDoneXmlDecoder.php:164ReplyDoneXmlDecoder.php:193ReplyDo not want to have an invalid access warning in this check. Replaced the expression with the PHPStorm-proposed str_starts_with() though.XmlDecoder.php:201ReplyPhpStorm was confused by the missing return type of method parseXml() some lines before. PhpStorm agrees now.XmlDecodingOptions.php:46ReplyNot exactly getting the problem here. From my POV it should keep the default values of $this->options unless overwritten explicitly by an incoming $options array key.XmlEncoder.php:139ReplyDoneXmlEncoder.php:274ReplyNot exactly sure as quite some time has passed since looking at it previously. The loose comparison matches the comparison in the original GeneralUtility::array2xml(), but i guess we can switch to strict comparison here. All unit tests still succeed.XmlEncoder.php:323ReplyDoneXmlEncoder.php:336ReplyDoneXmlEncoder.php:338ReplyDoneXmlEncodingOptions.php:67ReplyDone -
core-ci voted on patch set 49 V -1 01/21/22, 16:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22360
-
Alexander Nitsche uploaded patch set 50 01/21/22, 16:22
-
Alexander Nitsche commented on patch set 50 01/21/22, 16:27
XmlDecoder.php:171ReplyFrom my POV type safety is not important here, as the values of nodeType and the constants are all integer values and are not defined by the user but by PHP. -
core-ci voted on patch set 50 V +1 01/21/22, 16:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22361
-
Andreas Kienast voted on patch set 50 CR +1 01/24/22, 06:42
-
Christian Kuhn commented on patch set 50 01/24/22, 09:17
XmlEncoder.php:88I still wonder if we need this huge confusing $additionalOptions implementation in our generic encoder / decoder implementation: afair, ext:impexp is the only place that uses this huge bandwidth of things. Maybe we should have a generic implementation that is much more streamlined and easier, and ext:impexp then has an own solution that does all the additional magic? -
Alexander Nitsche commented on patch set 50 01/24/22, 09:23
XmlEncoder.php:88ReplyWhat about using this issue only to separate XML handling into own class with own code coverage and handling the 10 MB memory issue, and moving all further refactorings into follow-up issues? The $additionalOptions is already encapsulated, so we can easily separate it into a sublass of XmlEncoder or similar which is only used in EXT:impexp. -
Christian Kuhn commented on patch set 50 01/24/22, 09:33
XmlEncoder.php:88Replyok. Since the class is marked '@internal experimental', i'm fine with that: we're not creating api here that would be hard to change afterwards. -
Andreas Kienast commented on patch set 50 01/24/22, 09:47
XmlDecodingOptions.php:46ReplyDone -
Oliver Hader commented on patch set 50 01/24/22, 10:56
XmlDecoder.php:51I think `$options` should be a class member variable, defined in a constructor - this way passing options to internal methods can be avoidedXmlDecoder.php:134That's compatibility to `GeneralUtility::xml2arrayProcess`, correct? Maybe adding that as a comment...XmlDecoder.php:144Hm, hm... hmmm... that's basically only for `T3:T3FlexForms`, correct?XmlEncoder.php:275In should be ensured that `$value` does not contain any XML literals - currently it seems to be possible to leave the CDATA section and create a new node, e.g. with `$value = ']]><evil>whatever</evil><![CDATA['` In other words: `$value` needs to be handled as well for that scopeXmlEncoder.php:329So basically control-chars (x00-x1f) without TAB, NL, CR Seems x7f should be included as well (https://en.wikipedia.org/wiki/Control_character) -
Oliver Hader commented on patch set 50 01/24/22, 11:17
XmlEncoder.php:275ReplyThis is how PHP would do it: https://3v4l.org/Mho7j -
Alexander Nitsche commented on patch set 50 01/24/22, 13:50
XmlDecoder.php:51ReplyYes, could be. Leaving it as method argument underlines the immutable character of this class though. Having no objections though to change it, if still wanted.XmlDecoder.php:134Reply UnresolvedYes. I think this option is self-explanatory. If you want to explain it, please suggest a comment.XmlDecoder.php:144ReplyI don't know all use cases. Just looked at the previous implementation and migrated this functionality into the current implementation.XmlEncoder.php:275Reply UnresolvedThis is in line with the current implementation of GeneralUtility::array2xml(). What about collecting all changes of functionality in a follow-up issue? Including requiring tests.XmlEncoder.php:329Reply UnresolvedThis is in line with the current implementation of GeneralUtility::array2xml(). What about collecting all changes of functionality in a follow-up issue? Including requiring tests. -
Oliver Hader commented on patch set 50 V +1 CR +1 01/24/22, 16:45
Alex, thanks for your feedback. I agree this change basically shall be focussed on tackling the memory issue when processing XML data - other aspects in terms of "enhancements" would be a topic for TYPO3 v12 in general then. For testing in current `main` branch, probably having this fix applied is helpful: https://review.typo3.org/c/Packages/TYPO3.CMS/+/73139 -
Alexander Nitsche commented on patch set 50 01/24/22, 17:53
See #96635 for follow-up XML handling tasks. -
Benjamin Franzke commented on patch set 50 01/25/22, 13:24
XmlDecoder.php:144ReplyHm.. but this has the potential of breaking CDATA sections that may contain <foo:bar> as content. It would always be returned as <foo___bar> as reactivateNamespaceInNodeNames() only works on parsed node names. I think this functionality (to parse invalid XML) should be opt-in. -
Stefan Bürk commented on patch set 50 CR +1 01/25/22, 13:24
XmlDecoder.php:51ReplyI would not move that to a class member variable on instanciating, as this would introduce state to this class. Having it passed through make this stateless and an instance reusable without need of cleanup (the option state). -
Alexander Nitsche commented on patch set 50 01/25/22, 13:35
XmlDecoder.php:51Reply UnresolvedAckXmlDecoder.php:144ReplyAck -
Alexander Nitsche uploaded patch set 51 01/25/22, 15:31
-
core-ci voted on patch set 51 V +1 01/25/22, 15:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22529
-
Alexander Nitsche commented on patch set 51 01/25/22, 15:43
XmlDecoder.php:144ReplyThank you very much for this finding! -
Alexander Nitsche uploaded patch set 52 01/25/22, 16:37
-
Alexander Nitsche commented on patch set 52 01/25/22, 16:40
Last optimization: Run expensive XML node name namespace decoding only if required. -
core-ci voted on patch set 52 V +1 01/25/22, 16:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22537
-
Andreas Kienast voted on patch set 52 CR +1 01/26/22, 09:21
-
Benjamin Franzke uploaded patch set 53 01/26/22, 09:30
-
Benjamin Franzke commented on patch set 53 01/26/22, 09:31
XmlDecoder.php:144ReplyI really meant as OPT-IN XmlDecoderOption, because this is invalid XML that we allow here. Pushed in patchset 53. -
core-ci voted on patch set 53 V +1 01/26/22, 09:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22557
-
Oliver Hader voted on patch set 53 CR +1 01/26/22, 12:05
-
Alexander Nitsche uploaded patch set 54 01/26/22, 12:38
-
core-ci voted on patch set 54 V +1 01/26/22, 12:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22561
-
Benjamin Franzke commented on patch set 54 01/26/22, 16:02
I know, you'd like to get this merged sooner rather than later, still I want to talk about the naming. Ignore it if you like, I won't object merging as is (as it's marked internal, but maybe we can tackle this late then), but I won't vote +1 with the current naming by myself. While reviewing I stumbled over the names XmlEncoder and XmlDecoder over and over again. This is very generic, and most of the code is handling with the very special TYPO3 XML data serialization aspects, so I'm included to make all of that custom options (as done in patchset 53) in order to justify the generic name. Now, that would probably ruin the readability as everything is then scoped into options, but it reveals one detail: This XML is in effect an own serialization format, not generic XML. Yes, I know, GeneralUtility::xml2array() used a generic name as well and never used a proper name, but I'd love if we could name this less generic. Naming aspects: * This is more of a serialization than encoding * It is TYPO3 specific (I guess no one else would automatically convert index=%d to an integer-based array without even using a xml namespace) * It is a specifc XML Dialect/Schema/Format * The utils are mostly used for FlexForm XML or TYPO3 Record Documents (Import/Export) Naming Ideas: 1) Typo3XmlFormatEncoder Typo3XmlFormatDecoder 2) Core\Serialization\Typo3XmlSerializer Core\Serialization\Typo3XmlParser 3) Core\Serialization\ArrayToTypo3XmlSerializer Core\Serialization\Typo3XmlToArrayParser 4) Introduce an acronym like… TDDL (TYPO3 Data Description Language), TDSL (TYPO3 Data Serialization Language), TXSL (TYPO3 XML Serialization Language) or TXSF (TYPO3 XML Serialization Format) …and use that instead of "Xml" in the name. But that's probably not worth it, as this format isn't soo super brilliant that we actually want to create an acronym here. I personally would like to see option 2) What's you opinion, did you also already thought about naming options? -
Alexander Nitsche commented on patch set 54 01/26/22, 16:35
Reply
Sure, let's add the renaming to the follow-up patch collection #96635. I personally would opt for 2b) Core\Serializer\Typo3XmlSerializer Core\Serializer\Typo3XmlParser -
Alexander Nitsche commented on patch set 54 02/28/22, 09:01
Reply
Benjamin, as nobody else seems to want to throw in a new FQCN: Are you preferring (2) Core\Serialization\Typo3XmlSerializer Core\Serialization\Typo3XmlParser or (2b) Core\Serializer\Typo3XmlSerializer Core\Serializer\Typo3XmlParser ? -
prospero commented on patch set 54 04/05/22, 11:58
Hello everyone, I just wanted to express my gratitude for all the hard work, that has been done so far to fix this issue. For me as a user the 10 MB limit to the imports of the impexp-extension is less of an abstract problem, but a very real and serious one. My work centers around data migration and it looks like I will soon have to tunnel hundreds of sites, some coming in XML-files beyond the 100 MB mark in size, through the impexp-extension into a TYPO3 v10 (there is no upgrade to v11 in sight on my employer's systems). The reasons why I have to go this way would take too long to explain here, but everything depends on the impexp-Extension. I have an emergency backup-plan where I subdivide the page trees into bits under 10 MB, generating unique UIDs and enforcing them for cross-part-import-consistency, but this -being a one-person-job- might result in hundreds of hours of data-loss-risk-prone, monotonous manual imports. Thus I've been following the news about this issue for 7 months with great hopes. Having this bug-fix included in one of the sooner updates for v10 or at least as a patch would be amazing. I am so looking forward to this. All the best! =) -
Andreas Kienast uploaded patch set 55 04/07/22, 06:44
-
core-ci voted on patch set 55 V -1 04/07/22, 06:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/25708
-
Andreas Kienast uploaded patch set 56 04/07/22, 08:30
-
core-ci voted on patch set 56 V -1 04/07/22, 08:46 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/25721
-
Benni Mack uploaded patch set 57 04/16/22, 11:13
-
core-ci voted on patch set 57 V +1 04/16/22, 11:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26088
-
Susi Moog commented on patch set 57 04/18/22, 08:29
So, I tested this change with various scenarios this morning, and in general it makes the situation better. However, there is one scenario we should be aware of: The export (especially on command line) will export even very large websites without complaint - for example, my biggest test this morning is an 11GB export file. But even with this change here, there will be no possibility to import these files again, as there is still one hard limit in place - PHP_MAX_INT or: https://github.com/php/php-src/blob/master/ext/dom/document.c#L1320 IMHO we need two things here: a warning (or even a hard block) on the export side if the user exports something that cannot be imported again (because I as user would expect that something I exported with impexp can also be imported again, especially on the same system) and probably long-term a different strategy for exporting / importing large data sets (chunking, streaming, ...). The patch here mitigates the problem for "some" file sizes, but not all of them. -
Benjamin Franzke commented on patch set 57 05/04/22, 14:33
Reply Unresolved
2, so `Serialization` instead of `Serializer` (as in the examples I gave above). -
prospero commented on patch set 57 05/11/22, 10:45
Hello everyone, just an update from the "front-lines" (aka the user experience): I currently have to cross migrate a huge site from an entirely different CMS into TPYO3 v10. I tried to subdivide the generated import data. Unfortunately there is one subtree my algorithm fails to cut any smaller, its structure is too un-suited for that. I am already at 106 individual import-files and one is still 12+ MB in size. Not only will I have to manually import 100+ files and puzzle the site back together, but a central part won't make it. I also "look forward" to do that multiple times. For the final import, I'll have to lock out all editors for hours so I can enforce the UIDs and may not make a single mistake when doing that 100+ times. Sorry for whining, but this bottle neck makes me desparate. All the best -
Alexander Nitsche commented on patch set 57 05/30/22, 09:58
This patch is ready for merging. -
Alexander Nitsche commented on patch set 57 08/22/22, 13:10
This patch seems to be of no interest. It can be closed. -
Georg Ringer rebased onto patch set 58 08/23/22, 13:06
-
Georg Ringer voted on patch set 58 V +1 CR +1 08/23/22, 13:07
-
core-ci voted on patch set 58 V -1 08/23/22, 13:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/30379
-
Stefan Bürk uploaded patch set 59 08/23/22, 19:06
-
core-ci voted on patch set 59 V +1 08/23/22, 19:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/30387
-
Susi Moog voted on patch set 59 V +1 09/15/22, 07:31
-
Rémy DANIEL uploaded patch set 60
V +102/09/23, 23:41 -
Rémy DANIEL commented on patch set 60 02/09/23, 23:43
Rebased + fixed a bug in parsing array -
core-ci voted on patch set 60 V -1 02/10/23, 00:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/37512
-
Benni Mack rebased onto patch set 61
V -102/24/23, 15:39 -
core-ci voted on patch set 61 V -1 02/24/23, 15:47 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38063
-
Andreas Kienast uploaded patch set 62
V -102/25/23, 14:50 -
core-ci voted on patch set 62 V +1 02/25/23, 14:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38098
-
Benni Mack voted on patch set 62 V +1 CR +1 03/22/23, 21:45
-
Benni Mack rebased onto patch set 63
CR +1V +103/22/23, 21:46 -
core-ci voted on patch set 63 V +1 03/22/23, 21:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/39421
-
Benni Mack commented on patch set 63 V +1 CR +1 03/22/23, 22:07
time to get this one in. -
Andreas Kienast voted on patch set 63 V +2 CR +2 03/24/23, 14:28
-
Andreas Kienast merged patch set 64 03/24/23, 14:28