Review main Merged

62926[BUGFIX] Run tests with configured error reporting in PHPunit

Benni Mack CR +2 V +2 Normal +25 −52 updated 02/14/20, 08:16

Commit message

SystemEnvironmentBuilder calls

 error_reporting(E_ALL & ~(E_STRICT | E_NOTICE | E_DEPRECATED));

which in turn is called on every test to setup the environment.
This basically enforces this exact error reporting and does not
allow to configure the error reporting in phpunit.xml AT ALL.

Until TYPO3 v9, this was covered in each BaseTestCase in
setUpBeforeClass() to re-add E_NOTICE to make all tests run
without notice.

The functionality in SystemEnvironmentBuilder was added
in TYPO3 v6.0 while refactoring TYPO3's Bootstrap to ensure
that no notices in ext_localconf.php or ext_tables.php
were breaking the output. However, nowadays, TYPO3's
error reporting is set up directly after LocalConfiguration.php
inclusion, making this logic obsolete.

The functionality is therefore moved to the Bootstrap at a later point,
which is called for Functional and Acceptance Tests.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Georg Ringer +1
    • Oliver Klee +1
    • Oliver Hader +1
  • Verified V +2 Done
    • Benni Mack +2
    • Georg Ringer +1
    • TYPO3com +1 CI

Issues 1

Files 4

Review log 2534

Show 69 earlier messages
  1. Benni Mack uploaded patch set 1 01/16/20, 10:02

  2. TYPO3com voted on patch set 1 V -1 01/16/20, 11:29 Completed build in 1h 11m 22s on Thu, 16 Jan, 11:29 AM Test Summary: 1 of 60932 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40014

  3. Benni Mack uploaded patch set 2 01/16/20, 12:47

  4. TYPO3com voted on patch set 2 V -1 01/16/20, 13:10 Completed build in 18m 07s on Thu, 16 Jan, 01:10 PM Test Summary: 70312 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40040

  5. Alexander Schnitzler commented on patch set 2 01/16/20, 14:45

    Bootstrap.php:422

    I very much dislike binding the error_reporting to the existence of a configured exception handler class. Can't we introduce an env var that disables the this failsafe error reporting level? That way we can already run the core more strictly for testing without the need to implement another exception handler.
  6. Benni Mack commented on patch set 2 01/16/20, 15:02

    Bootstrap.php:422 Reply

    If we remove this, bamboo goes nuts :) Basically it is the same behaviour for the functional tests as before But it's a stupid idea, that's why it's a comment until we've made TYPO3 Core notice free.
  7. TYPO3com commented on patch set 2 01/16/20, 15:03 Completed build in 2h 29m 34s on Thu, 16 Jan, 03:03 PM Test Summary: 70338 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40040

  8. Alexander Schnitzler commented on patch set 2 01/16/20, 16:42

    Bootstrap.php:422 Reply

    I understand that we still need a very loose error reporting. What I don't get is why we move it to this position and make it conditional, depending on the existence of a registered exception handler class name.
  9. Benni Mack commented on patch set 2 01/16/20, 17:06

    Bootstrap.php:422 Reply

    - Before: SystemEnvironmentBuilder set it all the time, a later-on registered exceptionHandler handles all errors based on the TYPO3_CONF_VARS - After: We take the default php.ini. If the system configured with an exceptionHandler, it handles all errors based on TYPO3_CONF_VARS, if nothing is set, the error_reporting added here kicks in so people can still reach the installer (= same behavior as before)
  1. Alexander Schnitzler commented on patch set 2 01/16/20, 17:21

    Bootstrap.php:422 Reply

    I still don't get how the exception handler is involved in all this. If it's because we don't have one in the install tool, I'd rather override the error reporting in the bootstrap or install.php instead of making it dependent on a configuration value.
  2. Oliver Klee commented on patch set 2 01/18/20, 10:04

    Generally, I like running the tests with stricter error reporting at lot. I propose to move the cleanup part of this change to a separate change set (that should be merged first) to adhere to the "one kind of change per change set".
  3. Benni Mack commented on patch set 2 01/18/20, 15:28

    Bootstrap.php:422 Reply Unresolved

    Just rechecked this and agree that my solution is not the right way yet. This is how I see it: Bootstrap should keep setting error_reporting at any time (same instructions as before) - but do this before the exception handler is defined. The change would be to move the code from SystemEnvironmentBuilder to Bootstrap->initializeErrorHandling(). This would mean: - LocalConfiguration + AdditionalConfiguration to be notice free at any time (this would be different as before) - SystemEnvironmentBuilder would not set this, so Unit tests would not trigger this, but since functional + acceptance tests run the full bootstrap the error_reporting would be applied. My initial assumption was wrong - namely that ExceptionHandler would fiddle around with `error_reporting()` which is not the case.
  4. Benni Mack commented on patch set 2 01/18/20, 15:29

    Thanks for the feedback. > I propose to move the cleanup part of this change to a separate change set (that should be merged first) to adhere to the "one kind of change per change set". Ack!
  5. Benni Mack commented on patch set 2 01/29/20, 14:06

  6. Benni Mack uploaded patch set 3 01/30/20, 07:02

  7. TYPO3com voted on patch set 3 V -1 01/30/20, 07:17 Completed build in 14m 45s on Thu, 30 Jan, 07:17 AM Test Summary: 12 of 71272 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40342

  8. Oliver Klee commented on patch set 3 01/30/20, 08:24

    It also looks like there are some (possible new) warnings left to fix first.

    Bootstrap.php:370 Unresolved

    Do we expect the method to be overridable? Otherwise, the call should use self:: instead of static::.
  9. Benni Mack uploaded patch set 4 01/30/20, 12:21

  10. TYPO3com voted on patch set 4 V -1 01/30/20, 12:32 Completed build in 11m 35s on Thu, 30 Jan, 12:32 PM Test Summary: 70822 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40354

  11. TYPO3com voted on patch set 4 V +1 01/30/20, 13:15 Completed build in 1h 05m 24s on Thu, 30 Jan, 01:15 PM Test Summary: 70822 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40354

  12. Oliver Hader commented on patch set 4 CR +1 01/30/20, 21:09

    CharsetConverter.php:453

    Unrelated... but `xdigit` instead of `alnum` ;-)
  13. Oliver Hader commented on patch set 4 01/30/20, 21:20

    CharsetConverter.php:453 Reply Unresolved

  14. Benni Mack uploaded patch set 5 01/31/20, 10:40

  15. Oliver Klee voted on patch set 5 CR +1 01/31/20, 10:53

  16. TYPO3com voted on patch set 5 V -1 01/31/20, 11:02 Completed build in 17m 55s on Fri, 31 Jan, 11:02 AM Test Summary: 12 of 71614 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40392

  17. Benni Mack rebased onto patch set 6 02/13/20, 10:59

  18. TYPO3com voted on patch set 6 V -1 02/13/20, 11:17 Completed build in 14m 50s on Thu, 13 Feb, 11:17 AM Test Summary: 12 of 71694 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-42

  19. Benni Mack uploaded patch set 7 02/13/20, 11:47

  20. Oliver Hader voted on patch set 7 CR +1 02/13/20, 12:06

  21. TYPO3com voted on patch set 7 V +1 02/13/20, 12:13 Completed build in 22m 23s on Thu, 13 Feb, 12:13 PM Test Summary: 71244 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-45

  22. Georg Ringer voted on patch set 7 V +1 CR +1 02/13/20, 12:40

  23. Oliver Klee voted on patch set 7 CR +1 02/13/20, 17:10

  24. Benni Mack voted on patch set 7 V +2 CR +2 02/14/20, 08:16

  25. Benni Mack merged patch set 8 02/14/20, 08:16