Review main Merged Breaking

86888[!!!][FEATURE] Support qualified and unqualified ISO8601 in DataHandler

Benjamin Franzke CR +2 V +2 Expert +808 −301 updated 03/29/25, 09:57

Commit message

Streamline DataHandler datetime field processing to support unqualified
ISO8601 dates instead of misusing fake UTC-0 ISO8601 for representing
LOCALTIME [1][2][3].

This allows DataHandler to correctly interpret offset information (like
a real UTC-0 datestring) and to perform proper transformation into the
respective database representation. (Original timezone information is
still normalized to UTC (integer fields) or localtime (native datetime
fields), but the point-in-time is correctly persisted.)

The TYPO3 backend date picker element is adapted to use unqualified
ISO8601 LOCALTIME as interchange format between FormEngine PHP and
HTML/JS code.

Why use localtime without offset information (=unqualified)?
------------------------------------------------------------

Conceptually this is not a change. It is a technical cleanup of the
current approach as we have been effectively using ISO8601 localtime
as interchange format between FormEngine and DataHandler before as well:

A faked ISO8601 UTC0 string was used that was interpreted as
localtime. For example 1999-11-11T11:11:11Z on a server timezone
Europe/Berlin was interpreted as:
1999-11-11T11:11:11+01:00
           ^^

Formally seen that was wrong as 1999-11-11T11:11:11Z actually refers to
the 12th hour in Germany, when taking the time zone database rules for
Europe/Berlin in November 1999 (=CET with offset +01:00) into account:
1999-11-11T12:11:11+01:00
           ^^

That means we actually misused UTC offset Z(ero) als localtime qualifier,
which is wrong because it refers to UTC+0 and thus formally to another
point in time that we actually tried to encode.

As a consequence we had to subtract the servers current timezone offset
after parsing the fake UTC0 date string leading to complex code and
misunderstandings which introduced bugs like #103693.

Using explicit localtime basically allows to remove all these
workarounds and results in much simpler code.

Why not switch to real UTC + offsets directly?
----------------------------------------------

Editors (currently) do only enter dates, no timezone offsets or
timezone names.
They are supposed to enter dates in the servers timezone (which is
currently not introspectable from the backend UI, but that's another
topic and a UX issue).

To transform the raw user input into the server timezone it's important
to know what the *user intended* date and time is, that means we need to
know the date and time values the user has selected and not a mapped
representation in UTC, otherwise the JS datepicker code would need to be
aware of the server timezone including timezone rules.

That's possible but actually a major task. A simple "current" timezone
offset (like +01:00 for Europe/Berlin during winter time) would not be
sufficient, since the datepicker would need to be aware of future
daylight saving times and adjust offsets in order to create a correct
UTC-offset for the selected time to match the server timezone (for
example when picking a datetime from the summer time range while it's
currently winter time).

Therefore, as long as we do not expose a timezone selection to the
FormEngine frontend, we strive for the simplicity of raw user input and
perform the transformation into a date + timezone on the server.

This approach has also been chosen to be used by WHATWG[3][4] for
the `<input type="datetime-local">` element, where qualified
ISO8601 dates are explicitly forbidden, instead an unqualified
ISO8601 localtime date is expected.

Adopting to this paradigm will allow us to replace our custom datetime
picker with the native `<input type="datetime-local">` at some point.
(Currently we're not happy with the Firefox UX and therefore stay with
a custom JavaScript based date picker, but it's good to be prepared to
switch at any point in time by using a compatible interchange format).

That means: Switching to localtime now will pave the way for timezones
in UI later on, because we're than able to honor custom timezone
information as described in the example in [3] (airpoirt datetime-local
timezone <select> field).

API changes
-----------

Now that we adapt DataHandler to accept localtime for FormEngine to
DataHandler communication, we also officially support datetime offsets
to be provided for DataHandler-as-an-API usage, without the previously
quite "unexpected" result that timezone offsets were cropped,
because of our localtime misinterpretation.

That means, DataHandler now accepts five different *input* formats,
regardless of the persistence method of the respective field
(e.g. native datetime or timestamp field):

   * ISO8601 in LOCALTIME (Y-m-d\\TH:i:s):
     1999-11-11T11:11:11
     (interchange format between TYPO3 backend server/html/js)

   * ISO8601 with timezone offsets (Y-m-d\\TH:i:sP):
     1999-11-11T10:11:11Z
     1999-11-11T11:11:11+01:00
     (To be used as API)

   * SQL flavored date/times (Y-m-d H:i:s):
     1999-11-11 11:11:11
     1999-11-11
     11:11:11
     (To be used for copy operations of native datetime fields)

   * Unix timestamps (U):
     942315071
     (To be used for copy operations of unix timestamp fields)

   * DateTime objects with timezone offsets.
     NEW! Allows to pass-in `\DateTimeInterface` objects as-is without
     requiring a to-string conversion.
     // same effect as ISO8601 LOCALTIME above
     new \DateTimeImmutable('1999-11-11T11:11:11')
     // same effect as ISO8601 with offset above
     new \DateTimeImmutable('1999-11-11T11:11:11+00:00')
     new \DateTimeImmutable('1999-11-11T12:11:11+01:00')
     // Offsets of custom timezones will be accepted an converted to
     // UTC for timestamp persistence and
     // LOCALTIME for native datetime fields
     // Heads UP: That means timezone offset is lost on persistence,
     //           but the date in time is stored correctly, without
     //           cropping offsets
     new \DateTimeImmutable(
         '1999-11-11T11:11:11',
         new \DateTimeZone('Asia/Jerusalem')
     )

This change is "breaking" because the previous fake ISO8601 UTC0 strings
will now be interpreted as *real* UTC0 and the offset to the server
timezone will be normalized instead of being cropped.

That means: For those who used DataHandler as an API, adaptions will be
needed to remove previous "workarounds" for DataHandlers broken DateTime
handling.

[1] https://en.wikipedia.org/wiki/ISO_8601#Local_time_(unqualified)
[2] https://html.spec.whatwg.org/multipage/common-microsyntaxes.html#local-dates-and-times
[3] https://html.spec.whatwg.org/multipage/input.html#local-date-and-time-state-(type=datetime-local)
[4] https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/datetime-local

Review

  • Code review CR +2 Done
    • Benjamin Franzke +2
    • Andreas Kienast +1
    • Benni Mack +1
  • Verified V +2 Done
    • Benjamin Franzke +2
    • Andreas Kienast +1
    • Benni Mack +1
    • core-ci +1 CI

Issues 2

Files 31

Show 11 more files

Review log 5897

Show 4072 earlier messages
  1. Benjamin Franzke uploaded patch set 1 11/06/24, 06:02

  2. core-ci voted on patch set 1 V -1 11/06/24, 06:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69131

  3. Benjamin Franzke uploaded patch set 2 V -1 11/06/24, 06:14

  4. core-ci voted on patch set 2 V -1 11/06/24, 06:21 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69132

  5. Benjamin Franzke uploaded patch set 3 V -1 11/06/24, 12:19

  6. core-ci voted on patch set 3 V -1 11/06/24, 12:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69142

  7. Benjamin Franzke uploaded patch set 4 V -1 11/06/24, 12:30

  8. core-ci voted on patch set 4 V -1 11/06/24, 12:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69143

  9. Benjamin Franzke uploaded patch set 5 V -1 11/06/24, 12:49

  10. core-ci voted on patch set 5 V +1 11/06/24, 12:56 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69144

  11. Benjamin Franzke updated the commit message in patch set 6 V +1 11/06/24, 13:32

  12. core-ci voted on patch set 6 V +1 11/06/24, 13:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69145

  13. Benjamin Franzke uploaded patch set 7 V +1 11/07/24, 02:22

  14. core-ci voted on patch set 7 V -1 11/07/24, 02:28 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69158

  15. Benjamin Franzke uploaded patch set 8 V -1 11/07/24, 08:39

  16. core-ci voted on patch set 8 V +1 11/07/24, 08:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69172

  17. Benjamin Franzke uploaded patch set 9 V +1 11/07/24, 09:27

  18. core-ci voted on patch set 9 V -1 11/07/24, 09:34 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69173

  19. Benjamin Franzke uploaded patch set 10 V -1 11/07/24, 09:39

  20. core-ci voted on patch set 10 V +1 11/07/24, 09:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69174

  21. Benjamin Franzke uploaded patch set 11 V +1 11/07/24, 13:01

  22. core-ci voted on patch set 11 V +1 11/07/24, 13:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69186

  23. Benjamin Franzke uploaded patch set 12 V +1 11/07/24, 13:50

  24. core-ci voted on patch set 12 V -1 11/07/24, 13:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69193

  25. Benjamin Franzke uploaded patch set 13 V -1 11/07/24, 14:00

  26. Benjamin Franzke uploaded patch set 14 11/07/24, 14:05

  27. core-ci voted on patch set 14 V +1 11/07/24, 14:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69196

  28. Benjamin Franzke uploaded patch set 15 V +1 11/08/24, 05:47

  29. core-ci voted on patch set 15 V +1 11/08/24, 05:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69217

  30. Andreas Kienast commented on patch set 15 11/08/24, 07:22

    Germany
    a UX ("ju-eks")
    otherwise
    I'm confused, shouldn't this be `1999-11-11T12:11:11+01:00` here?
    Also here
  31. Benjamin Franzke updated the commit message in patch set 16 V +1 11/08/24, 07:30

  32. Benjamin Franzke commented on patch set 16 11/08/24, 07:30

    Reply

    Done

    Reply

    Done

    Reply

    Done

    Reply

    was only meant as an example, but I understand the confusion. yes, it is much better to point to the same date in time for all three examples.

    Reply

    Done
  33. core-ci voted on patch set 16 V +1 11/08/24, 07:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69220

  34. Benjamin Franzke rebased onto patch set 17 V +1 11/19/24, 22:45

  35. core-ci voted on patch set 17 V +1 11/19/24, 22:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69752

  36. Benjamin Franzke uploaded patch set 18 V +1 11/20/24, 06:33

  37. core-ci voted on patch set 18 V +1 11/20/24, 06:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69768

  38. Benjamin Franzke uploaded patch set 19 V +1 11/20/24, 21:12

  39. core-ci voted on patch set 19 V +1 11/20/24, 21:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69874

  40. Benjamin Franzke uploaded patch set 20 V +1 11/20/24, 21:43

  41. core-ci voted on patch set 20 V -1 11/20/24, 21:50 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69876

  42. Benjamin Franzke uploaded patch set 21 V -1 11/21/24, 07:48

  43. core-ci voted on patch set 21 V +1 11/21/24, 07:56 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69890

  44. Garvin Hicking commented on patch set 21 12/04/24, 13:07

    Can we relate #68651 to this as well, or is problematic timezone Extbase handling outside this scope?
  45. Benjamin Franzke commented on patch set 21 12/04/24, 13:29

    Reply

    Probably not related to this one as this is really about datahandler <=> formengine, but this could become part of a datetime epic (I don't want to promise too much though) and I think it is very important to finally fix that in extbase as well, yes.
  46. Andreas Kienast uploaded patch set 22 V +1 12/05/24, 15:07

  47. core-ci voted on patch set 22 V +1 12/05/24, 15:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/70662

  48. Benjamin Franzke uploaded patch set 23 V +1 12/08/24, 19:47

  49. core-ci voted on patch set 23 V -1 12/08/24, 19:53 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/70780

  50. Benjamin Franzke uploaded patch set 24 V -1 12/08/24, 20:01

  51. core-ci voted on patch set 24 V +1 12/08/24, 20:07 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/70781

  52. Benjamin Franzke marked it work in progress 12/19/24, 09:30

  53. Benjamin Franzke uploaded patch set 25 V +1 03/17/25, 09:14

  54. Benjamin Franzke uploaded patch set 26 03/17/25, 09:17

  55. core-ci voted on patch set 25 V +1 03/17/25, 09:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75701

  56. core-ci voted on patch set 26 V +1 03/17/25, 09:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75702

  57. Benjamin Franzke uploaded patch set 27 V +1 03/17/25, 09:48

  58. core-ci voted on patch set 27 V +1 03/17/25, 09:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75703

  59. Benjamin Franzke uploaded patch set 28 V +1 03/17/25, 10:44

  60. core-ci voted on patch set 28 V +1 03/17/25, 10:52 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75706

  61. Benjamin Franzke uploaded patch set 29 V +1 03/18/25, 10:07

  62. core-ci voted on patch set 29 V +1 03/18/25, 10:16 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75768

  63. Benjamin Franzke uploaded patch set 30 V +1 03/18/25, 11:02

  64. core-ci voted on patch set 30 V +1 03/18/25, 11:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75771

  65. Benjamin Franzke uploaded patch set 31 V +1 03/18/25, 11:13

  66. core-ci voted on patch set 31 V +1 03/18/25, 11:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75773

  67. Benjamin Franzke uploaded patch set 32 V +1 03/19/25, 13:35

  68. core-ci voted on patch set 32 V +1 03/19/25, 13:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75841

  69. Benjamin Franzke uploaded patch set 33 V +1 03/20/25, 05:49

  70. core-ci voted on patch set 33 V +1 03/20/25, 05:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75869

  71. Benni Mack commented on patch set 33 CR -1 03/20/25, 08:31

    DateTimeFormat.php:20

    I do not understand why this is an interface - we have no consumer for this interface - we only use the constant - the name of the interface does not follow our naming conventions
  72. Benjamin Franzke commented on patch set 33 CR -1 03/20/25, 08:35

    DateTimeFormat.php:20 Reply

    good points. It basically mimics the nomenclature of https://www.php.net/manual/de/class.datetimeinterface.php#datetimeinterface.constants.iso8601 – but you are right, it missed the annoying Interface suffix and `Interface` does not really makes sense here. Will fix.
  1. Benjamin Franzke uploaded patch set 34 CR -1 V +1 03/20/25, 10:28

  2. Benjamin Franzke commented on patch set 34 03/20/25, 10:28

    DateTimeFormat.php:20 Reply

    Done
  3. core-ci voted on patch set 34 V +1 03/20/25, 10:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75884

  4. Benjamin Franzke uploaded patch set 35 V +1 03/21/25, 06:10

  5. core-ci voted on patch set 35 V +1 03/21/25, 06:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75929

  6. Benjamin Franzke uploaded patch set 36 V +1 03/21/25, 07:25

  7. core-ci voted on patch set 36 V +1 03/21/25, 07:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/75938

  8. Benjamin Franzke uploaded patch set 37 V +1 03/27/25, 10:20

  9. core-ci voted on patch set 37 V +1 03/27/25, 10:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/76434

  10. Benni Mack updated the commit message in patch set 38 V +1 03/27/25, 12:19

  11. core-ci voted on patch set 38 V +1 03/27/25, 12:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/76450

  12. Benni Mack voted on patch set 38 CR +1 V +1 03/27/25, 12:32

  13. Benjamin Franzke rebased onto patch set 39 CR +1 V +1 03/27/25, 14:10

  14. Benjamin Franzke marked it ready for review 03/27/25, 14:12

  15. Benni Mack voted on patch set 39 CR +1 V +1 03/27/25, 14:13

  16. core-ci voted on patch set 39 V +1 03/27/25, 14:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/76468

  17. Andreas Kienast commented on patch set 39 03/27/25, 14:57

    requiring

    Filter.html:119

    We could use `{f:constant(name: '\TYPO3\CMS\Core\Domain\DateTimeFormat::ISO8601_LOCALTIME')}` as well to make the connection more clear.

    DataHandler.php:2031

    PhpStorm complains here: > Expected parameter of type '\DateTime', '\DateTimeInterface' provided

    Breaking-105549-ImprovedISO8601DateHandlingInTYPO3DataHandler.rst:34

    essence

    Breaking-105549-ImprovedISO8601DateHandlingInTYPO3DataHandler.rst:79

    shifting
  18. Andreas Kienast uploaded patch set 40 CR +1 V +1 03/27/25, 15:01

  19. Andreas Kienast commented on patch set 40 03/27/25, 15:01

    Reply

    Done

    Filter.html:119 Reply

    Done

    DataHandler.php:2031 Reply

    Done

    Breaking-105549-ImprovedISO8601DateHandlingInTYPO3DataHandler.rst:34 Reply

    Done

    Breaking-105549-ImprovedISO8601DateHandlingInTYPO3DataHandler.rst:79 Reply

    Done
  20. core-ci voted on patch set 40 V +1 03/27/25, 15:10 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/76474

  21. Benni Mack voted on patch set 40 CR +1 V +1 03/27/25, 15:12

  22. Andreas Kienast voted on patch set 40 CR +1 V +1 03/27/25, 15:22

  23. Benjamin Franzke voted on patch set 40 CR +2 V +2 03/27/25, 15:23

  24. Benjamin Franzke merged patch set 41 03/27/25, 15:23

  25. Lina Wolf commented on patch set 41 03/29/25, 09:57

    We need some support in documenting this, I dont know enough about this topic to do it myself: https://github.com/TYPO3-Documentation/Changelog-To-Doc/issues/1227