Review main Merged

47196[BUGFIX] Load XML files of Extension Manager properly

Andreas Kienast CR +2 V +2 Easy +4 −4 updated 05/17/16, 09:55

Commit message

Since the XEE security fix (I26701fc2ffb5aed7ccbd96c168aef571d012091e),
the XML files in the Extension Manager may are not loaded anymore, depending
on the machine. Change the way how the files are loaded to fix the issue.

Review

  • Code review CR +2
    • Oliver Hader +2
    • Mathias Schreiber +1
    • Christian Kuhn +1
    • Daniel Maier +1
    • Susi Moog +1
  • Verified V +2
    • Oliver Hader +2
    • Mathias Schreiber +1
    • Stephan Großberndt +1
    • Christian Kuhn +1

Issues 1

Files 2

Review log 45

Show 20 earlier messages
  1. Andreas Kienast uploaded patch set 1 03/11/16, 14:57

  2. Nicole Hummel commented on patch set 1 03/11/16, 15:03

    MirrorXmlPullParser.php:62

    Is this needed still than?
  3. Nicole Hummel commented on patch set 1 03/11/16, 15:04

    Error messages have to be changed as well -> error in parsing, not opening the file
  4. Andreas Kienast wrote 03/11/16, 15:12

    Patch set 2: Commit message was updated.
  5. Markus Klein wrote 03/11/16, 15:33

    Patch set 3: Published edit on patch set 2.
  6. Markus Klein commented on patch set 3 03/11/16, 15:35

    ExtensionXmlPullParser.php:57

    here we possibly have a performance problem. we now load the whole big xml into memory for sure. I don't know if ->open() would read from the file in smaller junks maybe.
  7. Nicole Hummel commented on patch set 3 03/11/16, 15:39

    ->open() doesn't but IMHO the ->read() does
  8. Nicole Hummel commented on patch set 3 CR +1 V +1 03/11/16, 17:32

    Have looked up for DOM, SimpleXML and XMLReader usage now

    ExtensionXmlPullParser.php:57 Reply

    Seems no other workaround here than loading the full content in memory.
  9. Nicole Hummel uploaded patch set 4 03/11/16, 19:53

  10. Markus Klein voted on patch set 4 CR +1 V +1 03/11/16, 20:17

  11. Nicole Hummel rebased onto patch set 5 03/12/16, 11:47

  12. Markus Klein voted on patch set 5 CR +1 V +1 03/13/16, 16:42

  13. Daniel Maier voted on patch set 5 CR +1 03/13/16, 19:25

  14. Jigal van Hemert commented on patch set 5 03/13/16, 22:07

    Question about the reason why it works, see comment

    ExtensionXmlPullParser.php:56

    Can't find the reason why xml() works and open() doesn't in this case. There is no libxml_disable_entity_loader() call here after the fresh parser is instantiated.
  15. Andreas Kienast commented on patch set 5 03/14/16, 14:52

    I can't reproduce this bug anymore after updating PHP 7 to PHP 7.0.4-5+deb.sury.org~trusty+1 (cli) ( NTS )
  16. Markus Klein commented on patch set 5 03/14/16, 15:24

    We had this issue on a PHP 5.6 machine.
  17. Oliver Hader commented on patch set 5 V -1 03/15/16, 12:39

    Cannot reproduce with current PHP 7.0 and PHP 5.6.
  18. Andreas Kienast commented on patch set 5 03/15/16, 12:44

    Seems like this was a bug in PHP that has been fixed with the latest updates.
  19. Oliver Hader commented on patch set 5 03/15/16, 12:52

    If added some notes to the forge issue - the warning you pointed out there has been discovered in 2011 already with TYPO3 4.6...
  20. Oliver Hader commented on patch set 5 03/15/16, 13:01

    * I've... (instead of "if" in my last message)
  1. Markus Klein commented on patch set 5 CR 0 V 0 03/15/16, 16:17

    Taking back my votes. Is there a link about the PHP bug? Would be interesting to see if that has been fixed in a wide range or if we still need this fix?
  2. Markus Klein commented on patch set 5 03/30/16, 23:24

    I can't reproduce the issue currently with my PHP versions. So what do we do about this one?
  3. Mathias Schreiber commented on patch set 5 CR +1 V +1 03/31/16, 10:06

    as demonstrated @sprint DUS
  4. Jigal van Hemert commented on patch set 5 CR -1 V -1 03/31/16, 14:46

    Can't reproduce; no documentation how to reproduce. There is no documentation of a bug that causes this behaviour. There is no explanation how this change causes the correct behaviour (the author said the fix was made by trial and error). If it's a known bug in PHP (or one of the libraries), is it already fixed in a later bugfix release? Should we in that case make a workaround or is it enough to update PHP or a library?
  5. Mathias Schreiber commented on patch set 5 03/31/16, 16:09

    as the commit message states: "depending on the machine". We have bug reports by users, we have 9 people sitting together seeing the bug in place. The patch fixes the bug. Here's what we do: Clean master clear opcode cache try to update either a language or the extension list. Every other click the error shows up. Apply the patch. Every click passes through and the system behaves as usual. I am not really happy with the -1 here. If we went down this road I'd -1 every single Windows patch because I'm unable to reproduce.
  6. Oliver Hader commented on patch set 5 03/31/16, 16:58

    Currently trying to reproduce and analyze the reasons
  7. Jigal van Hemert commented on patch set 5 03/31/16, 21:03

    > We have bug reports by users, we have 9 people sitting together > seeing the bug in place. > The patch fixes the bug. Can one or more of those nine people please explain *why* this effect occurs and *why* the patch fixes the issue? Both the open and xml function should work. I can't find any relevant bug reports on php.net. > I am not really happy with the -1 here. > If we went down this road I'd -1 every single Windows patch because > I'm unable to reproduce. It's not about "unable to reproduce"; it's more about "don't understand the problem" (and nobody can explain it so far).
  8. Markus Klein commented on patch set 5 03/31/16, 21:27

    @Mattes: I agree with Jigal here, we need to find the reason! Additionally this patch actually turns the PullParsers into nonsense. Those are meant to read the data piece by piece, especially because those can be really large. Reading the whole file now into memory is actually a HUGE impact and might kill the functionality of EM on even more systems. We could also drop the PullParsers completely then and only go with the PushParsers.
  9. Oliver Hader commented on patch set 5 CR +1 V 0 04/01/16, 09:44

    Removing my -1... I still cannot reproduce that, but have seen the behavior and fix on Mattes' system recently. I was not able to determine the reason after trying to reproduce with different vagrant boxes, Ubuntu VMs, analyzing the C code of PHP and LibXml2... It seems the bug exists in edge cases using a specific configuration...
  10. Wouter Wolters commented on patch set 5 04/06/16, 21:14

    Saw this today on a machine of us.
  11. Markus Klein commented on patch set 5 04/07/16, 07:06

    Wouter can you find out the exact version numbers of PHP libxml and all other xml related libs?
  12. Mathias Schreiber commented on patch set 5 04/07/16, 08:47

    Michiels box runs libxml 2.9.1
  13. Stephan Großberndt commented on patch set 5 V +1 04/07/16, 10:40

    We just had the same issue on a system of ours and were able to fix it with this patch.
  14. Stephan Großberndt commented on patch set 5 04/07/16, 10:44

    This is Debian 8 with PHP 5.6.17-0+deb8u1 and DOM/XML API Version => 20031129 libxml Version => 2.9.1
  15. Markus Klein commented on patch set 5 04/07/16, 11:47

    I run the same version on Fedora and do not have the issue. I'll collect that in the ticket now. We can therefore rule out libxml 2.9.1 for the time being
  16. Susi Moog commented on patch set 5 CR +1 04/11/16, 21:59

    as people obviously have this problem (saw that) and this patch fixes the problem (saw that, too) and nobody seems to be able to figure out why it happens we should for now merge this patch, as we need a solution here.
  17. Oliver Hader commented on patch set 5 CR +2 V +2 04/11/16, 22:14

    I'm all for adding this into the Core albeit I cannot reproduce it on different scenarios that I tried to rebuild. However, I've been one of Mattes' nine he mentioned some days ago. Besides that I measured the memory consumption with PHP7 with and without using file_get_contents - the differences are in the range of kiloBytes...
  18. Markus Klein commented on patch set 5 04/12/16, 05:53

    @Hader: Then we can simply change the \TYPO3\CMS\Extensionmanager\Utility\Parser\XmlParserFactory::$parsers to not load this class anymore. The *PushParsers are even better then than this implementation, as those do NOT load all content into memory.
  19. Markus Klein commented on patch set 5 04/12/16, 07:11

  20. Christian Kuhn voted on patch set 5 CR +1 V +1 04/12/16, 08:15

  21. Christian Kuhn commented on patch set 5 04/12/16, 08:22

    this one fixes an issue. this patch must be included in todays releases. questions regarding if we should drop one of the parsers should be answered with a later patch.
  22. Jigal van Hemert voted on patch set 5 CR 0 V 0 04/12/16, 08:39

  23. Oliver Hader commented on patch set 5 04/12/16, 08:45

    Thx Jigal!
  24. Gerrit merged patch set 5 04/12/16, 08:45

  25. Oliver Hader commented on patch set 6 05/17/16, 09:55