Review main Merged

86840[TASK] Introduce test of error-handling for FormEngine's basic-inputs

Ayke Halder CR +2 V +2 Hard +365 −0 updated 11/21/24, 11:43

Commit message

This introduces a multi-step test of the error-handling for
`FormEngineValidation`'s basic input fields.

Normal and unusual test-cases can be easily implemented via test-data
configuration on a element-by-element basis in the corresponding
`ElementsBasic*Cest`.

Review

  • Code review CR +2 Done
    • Andreas Kienast +2
    • Christian Kuhn +1
  • Verified V +2 Done
    • Andreas Kienast +2
    • Christian Kuhn +1
    • core-ci +1 CI

Issues 1

Files 4

Review log 2840

Show 915 earlier messages
  1. Ayke Halder uploaded patch set 1 10/30/24, 21:59

  2. core-ci voted on patch set 1 V -1 10/30/24, 22:07 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68869

  3. Ayke Halder uploaded patch set 2 V -1 10/30/24, 22:17

  4. Ayke Halder uploaded patch set 3 10/30/24, 22:19

  5. core-ci voted on patch set 2 V -1 10/30/24, 22:30 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68870

  6. core-ci voted on patch set 3 V -1 10/30/24, 22:31 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68872

  7. Ayke Halder uploaded patch set 4 V -1 10/30/24, 22:53

  8. core-ci voted on patch set 4 V -1 10/30/24, 22:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68874

  9. Ayke Halder uploaded patch set 5 V -1 10/30/24, 23:01

  10. core-ci voted on patch set 5 V +1 10/30/24, 23:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68876

  11. Andreas Kienast commented on patch set 5 CR -1 10/31/24, 09:58

    AbstractElementsBasicCest.php:58

    I do not like the naming here, `testJourney` is not explanatory. Also `$testInvalidData` implies that everything tested here is expected to be invalid, but then we have the `expectError` flag, contradicting the assumed purpose of the variable name.
  12. Ayke Halder uploaded patch set 6 CR -1 V +1 10/31/24, 17:15

  13. Ayke Halder commented on patch set 6 10/31/24, 17:20

    AbstractElementsBasicCest.php:58 Reply

    Acknowledged, the variables are renamed to better describe their function.
  14. core-ci voted on patch set 6 V +1 10/31/24, 17:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/68935

  15. Andreas Kienast uploaded patch set 7 V +1 11/19/24, 10:27

  1. Andreas Kienast voted on patch set 7 CR +1 V +1 11/19/24, 10:27

  2. core-ci voted on patch set 7 V +1 11/19/24, 10:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69683

  3. Benjamin Franzke commented on patch set 7 CR -1 11/19/24, 14:36

    ElementsBasicEmailCest.php:72

    Can we ideally merge this with `inputBehaviorVariants` in order to make clear what the actualy sequence of the test is.

    ElementsBasicEmailCest.php:73

    Optional: Also I feel like we actually want to test error-handling only, and additionally use `correct@example.com` to be able to cover this via `runInputFieldTest`. Maybe we should fork here and have a separate `testInputFieldSequence` (which only save's if actually requested)

    ElementsBasicEmailCest.php:75

    Alternative idea: `inputSequence` WDYT?
  4. Ayke Halder commented on patch set 7 11/19/24, 16:51

    ElementsBasicEmailCest.php:72 Reply

    There can not be an input-field with an open error at the end of the test-sequence - as this might result in follow-up tests not working as intended. How can we ensure that the last test-sequence of ~~inputBehaviorVariants~~ `'inputSequence'` always has `'expectError' => false` defined? Maybe implement something like the following code? ``` // In the end we try to save the tested record: so there is no error allowed for the last test in a test-sequence. $I->assertEquals( false, $inputSequence[array_key_last($inputSequence)]['expectError'], 'Last test in sequence is not allowed to expect an error: Implement a proper last test without an error.' ); ``` This one field-test defined outside of the test-sequence ensured that there is no error at the end of the test-sequence and the tested CE can be saved: * This ensures that the record has no 'hidden' problem anymore and really can be saved. * It also ensures that follow-up tests run in a clean environment.

    ElementsBasicEmailCest.php:73 Reply

    I'd like to really save any record after checking the error-handling sequence, to make sure that there are no 'hidden issues' left. One known 'hidden issue' is about all these browser-native input-field validations: Failing browser-native validation right now does not allow to save the record - even though there are absolutely no errors shown to the user.

    ElementsBasicEmailCest.php:75 Reply

    I'm fine with `inputSequence` too. (As long as we don't mix it up with `sequence test` or similar, as googling this turns up `sequential hypothesis testing` - which is not what our test implements here: https://en.wikipedia.org/wiki/Sequential_analysis )
  5. Andreas Kienast commented on patch set 7 11/20/24, 08:18

    ElementsBasicEmailCest.php:72 Reply

    > as this might result in follow-up tests not working as intended I don't quite understand this. Each new test (case) re-opens the form and starts from the beginning.
  6. Andreas Kienast uploaded patch set 8 CR +1 V +1 11/20/24, 09:45

  7. core-ci voted on patch set 8 V -1 11/20/24, 09:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69778

  8. Andreas Kienast uploaded patch set 9 V -1 11/20/24, 10:00

  9. core-ci voted on patch set 9 V +1 11/20/24, 10:07 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69779

  10. Ayke Halder commented on patch set 9 11/20/24, 13:39

    ElementsBasicEmailCest.php:72 Reply

    Your refactoring with patchset 9 looks good! https://review.typo3.org/c/Packages/TYPO3.CMS/+/86840/7..9 I'd like to implement the possibility to have the last test of an input-sequence failing (`expectedError = true`). I'll change one of the test-sequences to have the last test with `expectedError = true` and see what happens.

    ElementsBasicEmailCest.php:75 Reply

    This is changed to `testSequence` with patchset 9: https://review.typo3.org/c/Packages/TYPO3.CMS/+/86840/7..9
  11. Ayke Halder uploaded patch set 10 V +1 11/21/24, 10:16

  12. core-ci voted on patch set 10 V +1 11/21/24, 10:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69911

  13. Andreas Kienast commented on patch set 10 CR +1 V +1 11/21/24, 10:27

    ElementsBasicEmailCest.php:72 Reply

    Done

    ElementsBasicEmailCest.php:73 Reply

    Done
  14. Ayke Halder commented on patch set 10 11/21/24, 10:31

    I implemented an optional `comment` to better discern the separate test-sequences inside one data-provider. This implements full checks for all simple text field validations with `min`/`max`.

    ElementsBasicEmailCest.php:72 Reply

    I didn't find a proper example to naturally implement the case `expectedError = true` as a last test. Nevertheless I checked this case locally, and it is properly working.

    ElementsBasicEmailCest.php:73 Reply

    The check for save-ability is separate issue. So this can be implemented by an separate patch.
  15. Christian Kuhn commented on patch set 10 CR +1 V +1 11/21/24, 10:42

    let's go with it. as usual with ac tests, we'll see over time if they are stable enough already or if they need further finetuning.
  16. Andreas Kienast voted on patch set 10 CR +2 V +2 11/21/24, 10:42

  17. Ayke Halder rebased onto patch set 11 CR +1 V +1 11/21/24, 11:10

  18. Andreas Kienast commented on patch set 11 CR -1 11/21/24, 11:14

    Tests are now broken due to changes done in https://review.typo3.org/c/Packages/TYPO3.CMS/+/87136.
  19. Andreas Kienast uploaded patch set 12 CR -1 11/21/24, 11:23

  20. core-ci voted on patch set 11 V -1 11/21/24, 11:23 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69921

  21. Andreas Kienast uploaded patch set 13 11/21/24, 11:23

  22. core-ci voted on patch set 13 V +1 11/21/24, 11:38 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69927

  23. Christian Kuhn voted on patch set 13 CR +1 V +1 11/21/24, 11:43

  24. Andreas Kienast voted on patch set 13 CR +2 V +2 11/21/24, 11:43

  25. Andreas Kienast merged patch set 14 11/21/24, 11:43