Review main Abandoned

80389[TASK] Remove implicit wait timeout from acceptance tests

Benjamin Franzke CR 0 V 0 Normal +49 −17 updated 02/12/24, 12:46

Commit message

Codeception was configured to implicitly wait up to 2 seconds
when searching for a specific element in order for assertions
to become true.
This lead to flaky tests, whenever this 2-sec limit has been
reached from time to time.

Furthermore the distinction between specific and unspecific
locators [1] caused difficutlies for developers to understand,
when an implicit wait is applied or not:

  $I->click('Login'); // implicit wait DISABLED, not a specific locator
  $I->click('#login'); // implicit wait ENABLED, locator is CSS ID

With moving to `wait: 0` we try to ensure not to introoduce
similar cases again, in order to always wait explicitly when
performing asynchronous operations.

[1] https://codeception.com/docs/AcceptanceTests#smartwait

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 11

Review log 1117

  1. Benjamin Franzke uploaded patch set 1 08/04/23, 11:24

  2. core-ci voted on patch set 1 V -1 08/04/23, 11:36 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45050

  3. Benjamin Franzke uploaded patch set 2 V -1 08/07/23, 19:43

  4. core-ci voted on patch set 2 V -1 08/07/23, 19:54 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45201

  5. Tymoteusz Motylewski commented on patch set 2 08/07/23, 20:54

    ContentElementsCest.php:45

    is adding this line really helping? The implicit wait feature is there to solve cases exactly like this without the need of having $I->waitForElementVisible(, followed by $I->see( with same selector. waitForElement afaik has 30s timeout. So why not increasing the global implicit wait from 2s to e.g. 5 or 10? or use the "retry" functionality https://codeception.com/docs/AcceptanceTests Other acceptance frameworks like codeception or playwright have built in implicit wait and retry.
  6. Benjamin Franzke uploaded patch set 3 V -1 08/08/23, 02:55

  7. core-ci voted on patch set 3 V -1 08/08/23, 03:06 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45210

  8. Benjamin Franzke uploaded patch set 4 V -1 08/08/23, 03:42

  9. core-ci voted on patch set 4 V +1 08/08/23, 03:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45212

  10. Benjamin Franzke commented on patch set 4 08/08/23, 04:12

    ContentElementsCest.php:45 Reply

    Thanks for your insight. I agree that could be an alternative, but: There are two reasons why I wanted to try this change: a) It seems most TYPO3 developers (that I talked to) did not actually know that there is an implicit wait configured/happending all over the place, so they couldn't understand, why test fail from time to time. (It took me hours to find about that myself and others never did and often added TODOs/comments that showcased, that codeception was a magic blackbox, which caused some wild guessing workarounds in code, often simply adding 2seconds of timeouts, which then succeed by luck but actually didn't change anything, as the replicated the implicit timeout) b) It don't think it is obvious, which locators perform implicit wait, and which not. Seems like one would actually need to have a cheat-sheet open all the time, that lists the implicit wait scenarios. Please see the documentation. https://codeception.com/docs/AcceptanceTests#smartwait Like $I->see('Hello, Davert'); // DISABLED, Not a locator Now you say this one implicitly waits: $I->see('TYPO3 Styleguide Frontend', '.content'); Cool, so `see()` becomes an implicit waiter when gloal wait *AND* a second parameter is supplied. This is hard to understand for those that do not work with codeception on a daily basis. So, I think, with playwright, where implict wait actually checks for things we need (like "can it receive events, or do I need to wait until it is no longer oscured by another element") and is builtin and not an afterthought, we would proably be fine with the automagic, but with codeception I'd propose this one, in order to be able to migrate to other tools from a very explicit set of tests. (There were some ideas, and I think this helps in case anyone wants to try out migrating, without having to figure out, why certain tests worked in codeception and do not when migrating).
  11. Benjamin Franzke updated the commit message in patch set 5 V +1 08/08/23, 08:11

  12. core-ci voted on patch set 5 V -1 08/08/23, 08:21 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45222

  13. Benjamin Franzke uploaded patch set 6 V -1 08/08/23, 09:23

  14. core-ci voted on patch set 6 V -1 08/08/23, 09:35 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/45228

  15. Christian Kuhn commented on patch set 6 08/08/23, 09:59

    ContentElementsCest.php:45 Reply

    Agree with Benjamin: The implicit wait is a black box nobody understands. We should get rid of it and instead plaster our tests with explicit waitForX() things, which we're doing at tons of places already, anyways. The fact that the implicit wait kick in only "sometimes" just adds confusion and does not help us. +1 for the patch approach. And yes, we need to have stable codeception tests before we walk away from it - there is some progress, we'll see what happens next weeks.
  16. Tymoteusz Motylewski commented on patch set 6 08/08/23, 10:47

    ContentElementsCest.php:45 Reply

    Thanks Benjamin for explanation, now I see the issue - the smart wait is only working when selector is passed. I'm ok with the change. Anyway, are there plans to use sth more modern like cypress or playwright? I saw Benji did some poc https://review.typo3.org/c/Packages/TYPO3.CMS/+/75520
  17. Benjamin Franzke abandoned the change 02/12/24, 12:46

    abandoned for the time being