66670[BUGFIX] Set timezone on TCA dbType input
Commit message
Review
-
Code review CR +2 Done
- Benjamin Franzke +2
- Markus Klein +1
-
Verified V +2 Done
- Benjamin Franzke +2
- Markus Klein +1
- TYPO3com +1 CI
Issues 1
Files 2
Review log 3947
Show 1822 earlier messages
-
Imko Schumacher uploaded patch set 1 11/17/20, 18:31
-
TYPO3com voted on patch set 1 V +1 11/17/20, 18:45 Completed build in 13m 13s on Tue, 17 Nov, 06:44 PM Test Summary: 73719 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7380
-
Martin Kutschker voted on patch set 1 CR +1 11/17/20, 19:15
-
Markus Klein commented on patch set 1 CR -1 11/17/20, 19:17
This is a highly critical area, which caused masses of regressions and frustration in the past. I would really appreciate if you could add proper test coverage with your change here. -
Markus Klein commented on patch set 1 11/17/20, 19:18
Please also note that we use real names for patch authors. So please fix that too. -
Martin Kutschker commented on patch set 1 CR 0 11/17/20, 21:01
> Patch Set 1: Code-Review-1 > > This is a highly critical area, which caused masses of regressions and frustration in the past. > I would really appreciate if you could add proper test coverage with your change here. Oh, there are currently no tests for this. Oops. -
Imko Schumacher marked it work in progress 11/17/20, 22:04
-
Imko Schumacher uploaded patch set 2 11/19/20, 17:44
-
Imko Schumacher commented on patch set 2 11/19/20, 17:45
Add unit tests for dbtype inputs -
Imko Schumacher uploaded patch set 3 11/19/20, 17:50
-
TYPO3com voted on patch set 2 V +1 11/19/20, 17:56 Completed build in 11m 43s on Thu, 19 Nov, 05:56 PM Test Summary: 73761 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7429
-
TYPO3com commented on patch set 3 11/19/20, 18:03 Completed build in 12m 28s on Thu, 19 Nov, 06:03 PM Test Summary: 73761 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7430
-
TYPO3com commented on patch set 3 11/19/20, 18:12 Completed build in 13m 38s on Thu, 19 Nov, 06:12 PM Test Summary: 73760 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7432
-
Markus Klein commented on patch set 3 11/20/20, 07:54
> Please also note that we use real names for patch authors. So please fix that too. -
Markus Klein commented on patch set 3 11/20/20, 08:00
DataHandler.php:1707I know this is probably not trivial, but could you check if this can be covered with a (functional?) test as well please? I'm not sure what may rely on this null value later on in the code. -
Imko Schumacher commented on patch set 3 11/20/20, 14:47
> Patch Set 3: > > > Please also note that we use real names for patch authors. So please fix that too. This issue is outside my responsibility, because my name is on my.typo3 and forge correct. I am in contact with Stephan Grossberndt "but [he] was unable to fix it easily". If you have any idea, feel free to contact me. -
Imko Schumacher wrote 11/20/20, 14:47
Removed reviewer Martin Kutschker. -
Imko Schumacher wrote 11/20/20, 14:47
Removed reviewer TYPO3com. -
Markus Klein commented on patch set 3 11/20/20, 14:52
> Patch Set 3: > > > Patch Set 3: > > > > > Please also note that we use real names for patch authors. So please fix that too. > This issue is outside my responsibility, because my name is on my.typo3 and forge correct. > I am in contact with Stephan Grossberndt "but [he] was unable to fix it easily". > If you have any idea, feel free to contact me. I'm sorry. I did not realize that the patch author is already correct! The Gerrit UI obviously does not display this information anymore, so I only saw that when I cherry-picked the patch. Thanks, all good then from the review-side. (The name in Gerrit does not matter that much then. But of course it's good to know whether one can change it or not.) -
Imko Schumacher commented on patch set 3 11/20/20, 14:58
Readd reviewers removed by mistake -
Markus Klein commented on patch set 3 11/20/20, 15:00
> The Gerrit UI obviously does not display this information anymore, so I only saw that when I cherry-picked the patch. Correcting myself: It is shown, when opening the detail view of the commit message... learned something new today. -
Imko Schumacher commented on patch set 3 11/20/20, 15:19
> Patch Set 3: > > (1 comment) (see reply in code)DataHandler.php:1707ReplyI do not think so, because an empty input value gets checked in Line 1675, where $value gets set to null. So this case only is true in following cases ($res['value']): - null: is already stored, no need to redefine - array(): returned by checkValue_input_Eval if 'required' is set - makes no sense to handle required on dbtype differently than on other types - 0: is a VALID (=BUG in previous version) value (e.g. unix timestamp for 00:00 / 1970-01-01T00:00:00Z) and should not be set to null, because it blocks e.g. range eval. - false, "0", , "": only occour on improper eval use / not at all In all other cases the value $res['value'] get sets in checkValue_input_Eval Line 2601. Because of this I think this can be removed without a problem and no test is needed.
-
TYPO3com commented on patch set 3 11/20/20, 15:46 Completed build in 21m 04s on Fri, 20 Nov, 03:46 PM Test Summary: 32403 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7475
-
Markus Klein commented on patch set 3 11/20/20, 20:18
DataHandler.php:1707ReplyHm... this code was explicitly added with a bugfix https://review.typo3.org/c/Packages/TYPO3.CMS/+/51296/ Unfortunately no test has been added to cover this case. -
Markus Klein commented on patch set 3 11/20/20, 20:20
DataHandler.php:1707Reply> 0: is a VALID (=BUG in previous version) value No this is an invalid value for a native datetime field! Neither 0 nor 0000-00-00 are valid values. -
Markus Klein voted on patch set 3 CR -1 11/20/20, 20:20
-
Imko Schumacher commented on patch set 3 11/21/20, 12:01
Since this is getting more than the original patch (the timezone problem), I think it is the best idea to outsource this into a new issue / patch. (I will do that later)DataHandler.php:1707Reply UnresolvedAll valid native date/time values get converted to a timestamp in Line 1685. "Convert the date/time into a timestamp for the sake of the checks" So in this part of the code we expect either a timestamp value or null (for invalid or default value) and not a native string. Example for a valid 0: If you enter 00:00 in the 'inputDateTime', the value "1970-01-01T00:00:00Z" gets submitted. This value gets converted in Line 1684/1685 to a DateTime Object and then to the timestamp 0. At this point I see a valid 0. At the back-conversion in Line 1740 it should then not use the default value (I just see an other bug there). Since this is getting more than the original patch (the timezone problem), I think it is the best idea to outsource this into a new issue / patch. (I will do that later) -
Imko Schumacher uploaded patch set 4 11/21/20, 12:09
-
Imko Schumacher uploaded patch set 5 11/21/20, 12:10
-
TYPO3com voted on patch set 4 V +1 11/21/20, 12:20 Completed build in 10m 55s on Sat, 21 Nov, 12:20 PM Test Summary: 32405 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7501
-
TYPO3com voted on patch set 5 V +1 11/21/20, 12:23 Completed build in 12m 31s on Sat, 21 Nov, 12:23 PM Test Summary: 32403 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-7502
-
Markus Klein commented on patch set 5 V +1 CR +1 11/21/20, 18:08
Thanks. Looks good to me. -
Johannes Schlier voted on patch set 5 V +1 CR +1 12/10/20, 11:52
-
Wouter Wolters voted on patch set 5 CR +1 12/10/20, 22:40
-
Benjamin Franzke commented on patch set 5 CR -1 12/18/20, 10:15
I probably miss something, but how does this change go in line with the following patch that was merged to master (thus will in v11 only): https://review.typo3.org/c/Packages/TYPO3.CMS/+/64053 That mentioned patch changed extbase to use local time instead of UTC, because the rest (of the core) uses local time as well. Now this patch changes Core to assume UTC, but extbase will use local time (in v11). Don't get me wrong, I personally think UTC would be fine in general, but as a master-only (probably breaking?) change, and that for all contexts, DataHandler and extbase... -
Markus Klein commented on patch set 5 12/18/20, 10:24
Reply
This is okay. The 64053 patch was not merged downwards, because it might be breaking for existing extensions that potentially have a workaround for this in in place. This patch is correct too. Both patches aim to have Core and Extbase in line with how Core currently (since 7.6 actually) is defined to work: int-DB fields (timestamp) => UTC native-DB fields => local time -
Benjamin Franzke commented on patch set 5 12/18/20, 14:58
Reply
Why does this patch parse the native-DB field as UTC, when you say that native-DB fields are supposed to be local time?Where is this condition reflected in the code? The value is now always parsed/created as DateTime with UTC timezone, no? -
Imko Schumacher commented on patch set 5 12/18/20, 16:30
Reply
We format the timestamp at the end with gmdate (formats in utc), because the frontend submits the value as strings with timezone information (see A). The second timezone argument is only relevant, if the input string has to timezone (see B). I see however the point, that the checks are done with the timestamp interpreted as utc date. However if the checks should be done in the local timezone, then a seperate patch only for master is needed (leading to more special cases / problems, e.g. time inputs). --- This function gets a string with a date, converts it to a timestamp (L 1684 f) to do some checks on it, and finally converts it back to a string (L 1740). This means, that the timezone is only in the processing of this function relevant and does not change, in which timezone something is stored. I know from two types as inputs: A: 2020-11-25T18:54:00Z: Submitted by the frontend with a inputDateTime control B: 2020-11-25 18:54:00: Copying an record in the backend For scenario A this patch changes nothing, since the string has timezone information in it (the timezone argument is then ignored). The end conversion uses gmdate (always utc timezone). For the scenario B this leads it to be parsed in the local timezone, however then be converted back in the utc timezone. The bugfix is for this case.Reply
No, since the timezone is only used, if the string does not contain it. "The $timezone parameter and the current timezone are ignored when the $datetime parameter either [...] or specifies a timezone (e.g. 2010-01-28T15:00:00+02:00)." (see https://www.php.net/manual/en/datetime.construct.php) -
Benjamin Franzke uploaded patch set 6 12/18/20, 21:00
-
Benjamin Franzke commented on patch set 6 12/18/20, 21:00
Reply
Thank you very much for your explanation. Well, gmdate() does *not* format in UTC, it formats depending on the local time. But that's "fine" taking Markus statements regarding native-DateTime fields into account (which are local time in TYPO3). It's just that the comment above that message is (and was) wrong. We don't store UTC timestamps in the database for dbType fields – that comment is actually the reason for my confusion, and also probably for yours. (So I'll adapt that..) (but) The real curlpit is: Frontend submits in local time, but marks that as "UTC" (by using the "Z" suffix as container). It would be better to *not* mark it, as it really is not UTC. Timezone is *unspecified*/local-time. Quoting the ISO 8601 spec: The zone designator is empty if use is made of local time in accordance with 4.2.2.2 through 4.2.2.4, it is the UTC designator [Z] if use is made of UTC of day in accordance with 4.2.4 and it is the difference-component if use is made of local time and the difference from UTC in accordance with 4.2.5.2. That's why this code is so ugly and buggy, we use UTC as a normalizing container (or in other words "please don't timeshift anything, we got a string based timestamp, but we actually just want a unix timestamp that we interpret in local time"), instead of *not* specifying a timezone at all. In my opinion the *real* fix would be to drop the "Z" suffix (if possible in JS) and parse the passed date in local time *always* (that means we would work with a DateTime object with local time instead of UTC). Than gmdate() would be replaced by a simple $dateTime->format().. On the other hand, your fix for this UTC-normalizing + gmdate() disaster is a good and unobtrusive fix, and are better than a massive rewrite for backporting this. Therefore I would be happy to merge this, but I feel like I really need to add some more comments to the code. So sorry for another roundtrip..Reply
Thank you, I wasn't aware of that. Probably worth a comment. -
TYPO3com voted on patch set 6 V +1 12/18/20, 21:09 Completed build in 09m 18s on Fri, 18 Dec, 09:09 PM Test Summary: 32508 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-8271
-
Markus Klein commented on patch set 6 12/18/20, 22:36
Reply
And JS is precisely the reason why we do that "Z-normalization" to communicate with the Backend-JS. We introduced this with v7 if I'm not mistaken to finally get rid of a multitude of other weird bugs. As far as I am aware, no real change has happened in the meantime in the JS-sphere in regards to time zone handling. Feel free to prove me wrong, which would indeed be the chance to get out Z-stuff removed again. -
Markus Klein voted on patch set 6 V +1 CR +1 12/18/20, 22:37
-
Imko Schumacher commented on patch set 6 12/19/20, 11:23
Thank you for the improvements in the commit message / comments. Just to clearify: gmdate() does not format the date in the local timezone but in the UTC timezone. "Identical to the date() function except that the time returned is Greenwich Mean Time (GMT). " (https://www.php.net/manual/en/function.gmdate.php) -
Benjamin Franzke commented on patch set 6 12/19/20, 13:28
Reply
Ack, you are right. -
Benjamin Franzke voted on patch set 6 V +2 CR +2 12/21/20, 12:43
-
Benjamin Franzke merged patch set 7 12/21/20, 12:43