Review main Merged

64152[TASK] Rework nightly test run to execute less tests

Anja Leichsenring CR +2 V +2 Hard +194 −101 updated 04/27/20, 17:53

Commit message

The current nightly run simply takes too long, we need to cut
down the sheer number of tests we run. The current setup
aims to execute a small subset of all tests on each DBMS and
Php version, but in total every test is run in each composer stage
at least once.

Review

  • Code review CR +2 Done
    • Anja Leichsenring +2
    • Markus Klein +1
  • Verified V +2 Done
    • Anja Leichsenring +2
    • Markus Klein +1
    • TYPO3com +1 CI

Issues 1

Files 2

Review log 3137

Show 812 earlier messages
  1. Anja Leichsenring uploaded patch set 1 04/13/20, 17:20

  2. Andreas Kienast commented on patch set 1 04/14/20, 03:47

    NightlySpec.java:41 Unresolved

    Wrong substitution
  3. Anja Leichsenring uploaded patch set 2 04/14/20, 15:52

  4. Anja Leichsenring wrote 04/16/20, 05:52

    Unset private
  5. Anja Leichsenring commented on patch set 3 04/16/20, 05:52

    Published edit on patch set 2.
  6. Anja Leichsenring commented on patch set 4 04/16/20, 05:53

    Published edit on patch set 3.
  7. Anja Leichsenring rebased onto patch set 5 04/16/20, 05:53

  8. Anja Leichsenring updated the commit message in patch set 6 04/16/20, 05:57

  9. TYPO3com voted on patch set 3 V +1 04/16/20, 06:09 Completed build in 16m 18s on Thu, 16 Apr, 06:09 AM Test Summary: 73571 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2624

  10. TYPO3com voted on patch set 4 V +1 04/16/20, 06:16 Completed build in 23m 23s on Thu, 16 Apr, 06:16 AM Test Summary: 73571 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2625

  11. TYPO3com voted on patch set 5 V +1 04/16/20, 06:21 Completed build in 27m 32s on Thu, 16 Apr, 06:21 AM Test Summary: 73571 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2626

  12. TYPO3com voted on patch set 6 V +1 04/16/20, 06:23 Completed build in 14m 15s on Thu, 16 Apr, 06:23 AM Test Summary: 73571 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2627

  1. Markus Klein commented on patch set 6 CR +1 04/16/20, 06:32

    looks good by reading
  2. Frank Nägler commented on patch set 6 04/16/20, 06:34

    I guess this is not, what we want. This patch remove the tests for possible PHP, composer, database combinations.+
  3. Anja Leichsenring commented on patch set 6 04/16/20, 06:40

    > Patch Set 6: > > I guess this is not, what we want. > This patch remove the tests for possible PHP, composer, database combinations.+ all combinations are still there, but on a subset of the tests, not all anymore.
  4. Alexander Schnitzler commented on patch set 6 04/16/20, 06:43

    Question: Shouldn't we first drop the least needed tests first? To me that's all the composer min stages. composer min is a theoretical construct. Nobody installs a site using minimum versions. And if you find any reason why people should do so, I'd rather have a different statagey having composer.json define the min version such as "foo/bar":"^4.5.17" instead of "^4.5.0". If this is to cut down the runtime, we should also consider simply throwing more agents on those runs. Politics aside here as I don't care how this would be accomplished but we have a problem that is caused by to few resources. Sacrificing tests doesn't feel like the best solution here.
  5. Markus Klein commented on patch set 6 04/16/20, 06:54

    > Patch Set 6: > > Question: Shouldn't we first drop the least needed tests first? To me that's all the composer min stages. composer min is a theoretical construct. Nobody installs a site using minimum versions. And if you find any reason why people should do so, I'd rather have a different statagey having composer.json define the min version such as "foo/bar":"^4.5.17" instead of "^4.5.0". > > If this is to cut down the runtime, we should also consider simply throwing more agents on those runs. Politics aside here as I don't care how this would be accomplished but we have a problem that is caused by to few resources. Sacrificing tests doesn't feel like the best solution here. I disagree here partially. Testing is always a trade-off between cost and benefit. It does not make sense to test scenarios over and over again, where faulty behaviour is expected to be seen only rarely. There we throw a lot of resources on a small problem. For instance: Running the same test in all PHP versions is in my POV a very expensive trade-off here. It should suffice to run things in the lowest and highest version. If some in between version suddenly fails, it is likely a bug in the current PHP release itself, otherwise it would have popped up already earlier. 100% certainty costs infinite money, which we all don't (and never will) have.
  6. Daniel Goerz commented on patch set 6 04/16/20, 06:57

    NightlySpec.java:109 Unresolved

    Looks like either the comment or the string is wrong.
  7. Anja Leichsenring commented on patch set 7 04/16/20, 06:59

    Published edit on patch set 6.
  8. TYPO3com voted on patch set 7 V +1 04/16/20, 07:13 Completed build in 14m 00s on Thu, 16 Apr, 07:13 AM Test Summary: 73571 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2632

  9. Alexander Schnitzler commented on patch set 7 04/16/20, 07:39

    > Patch Set 6: > > > Patch Set 6: > > > > Question: Shouldn't we first drop the least needed tests first? To me that's all the composer min stages. composer min is a theoretical construct. Nobody installs a site using minimum versions. And if you find any reason why people should do so, I'd rather have a different statagey having composer.json define the min version such as "foo/bar":"^4.5.17" instead of "^4.5.0". > > > > If this is to cut down the runtime, we should also consider simply throwing more agents on those runs. Politics aside here as I don't care how this would be accomplished but we have a problem that is caused by to few resources. Sacrificing tests doesn't feel like the best solution here. > > I disagree here partially. > Testing is always a trade-off between cost and benefit. > It does not make sense to test scenarios over and over again, where faulty behaviour is expected to be seen only rarely. There we throw a lot of resources on a small problem. > > For instance: Running the same test in all PHP versions is in my POV a very expensive trade-off here. It should suffice to run things in the lowest and highest version. If some in between version suddenly fails, it is likely a bug in the current PHP release itself, otherwise it would have popped up already earlier. > > 100% certainty costs infinite money, which we all don't (and never will) have. Yes, I agree with the PHP versions. That would be a good thing to change with very little impact on the testing but much impact on the execution time. What do you think about composer min?
  10. Markus Klein commented on patch set 7 04/16/20, 08:55

    > > What do you think about composer min? If I remember correctly this was added because of issues in the past. People added other dependencies which required a lower version of a package than what we tested on Bamboo. So things fell apart as the older version obviously was never properly tested. To be sure the currently defined minimum requirements actually work, those tests have been added. As far as I know this is a general recommendation for automated testing to cover lowest and highest versions.
  11. Markus Klein voted on patch set 7 CR +1 04/16/20, 08:55

  12. Alexander Schnitzler commented on patch set 7 04/16/20, 12:50

    > Patch Set 7: > > > > > What do you think about composer min? > > If I remember correctly this was added because of issues in the past. People added other dependencies which required a lower version of a package than what we tested on Bamboo. > So things fell apart as the older version obviously was never properly tested. > To be sure the currently defined minimum requirements actually work, those tests have been added. As far as I know this is a general recommendation for automated testing to cover lowest and highest versions. Ok, makes sense.
  13. Anja Leichsenring commented on patch set 7 04/16/20, 14:58

    hi guys, thanks for all the input, but this actually will not resolve the problem that the current infrastructure is not able to keep up with demand. So in order to get a step closer to reliably running tests, I propose this shortening. What other runs we drop (or not) could be the next step of the process and we can take that to Slack or the next ticket. What do you think?
  14. Alexander Schnitzler commented on patch set 7 04/17/20, 14:42

    There is one reason I stopped reviewing: I didn't really understand the exact outcome of this patch by just reading the commit message. The message is quite vague and I would like to see more details, maybe in the form of a list. Something like in these examples: * Runs only lowest and highest php versions * Does not execute MSSql with composer min/max any more
  15. Anja Leichsenring commented on patch set 7 04/17/20, 14:56

    > Patch Set 7: > > There is one reason I stopped reviewing: > I didn't really understand the exact outcome of this patch by just reading the commit message. > The message is quite vague and I would like to see more details, maybe in the form of a list. > > Something like in these examples: > > * Runs only lowest and highest php versions > * Does not execute MSSql with composer min/max any more good point, I will add such an overview.
  16. Anja Leichsenring commented on patch set 7 04/19/20, 06:50

    the matrix is in the ticket, if someone wants to take a look.
  17. Markus Klein commented on patch set 7 CR 0 04/19/20, 20:40

    Thanks Anja. I'd like to suggest one change: each php version should have a run for F+ and F-. background: some libs ship different packages(-versions) varying php versions.
  18. Anja Leichsenring commented on patch set 7 04/21/20, 05:33

    > Patch Set 7: -Code-Review > > Thanks Anja. I'd like to suggest one change: each php version should have a run for F+ and F-. > background: some libs ship different packages(-versions) varying php versions. will do, good point.
  19. Anja Leichsenring uploaded patch set 8 04/22/20, 16:34

  20. TYPO3com voted on patch set 8 V +1 04/22/20, 16:46 Completed build in 11m 44s on Wed, 22 Apr, 04:46 PM Test Summary: 73657 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2987

  21. Anja Leichsenring commented on patch set 8 04/24/20, 11:42

    the run as defined by the matrix in the ticket takes roughly 3 hours to complete.
  22. Simon Gilli commented on patch set 8 04/24/20, 12:13

    AbstractCoreSpec.java:1499

    Why not using an else if directly here?
  23. Markus Klein voted on patch set 8 V +1 CR +1 04/24/20, 12:22

  24. Anja Leichsenring commented on patch set 8 V +2 CR +2 04/27/20, 17:53

    AbstractCoreSpec.java:1499 Reply Unresolved

    maybe next time. This does not need to optimized, it is run only once.
  25. Anja Leichsenring merged patch set 9 04/27/20, 17:53