Review main Merged

69402[BUGFIX] Respect query string for regexp redirects

Guido Schmechel CR +2 V +2 Hard +390 −24 updated 10/04/21, 16:47

Commit message

Backend redirect record edit form was not clear about the
fact that regular expression do not respect query parameter
if selected too. RegExp matching has been done only with
the path, even if the request has query parameters.

Thus it was not obvious to users why regular expression
evaluation against query parameter part has not been
matched.

To mitigate this regular expression redirects are matched
against path and query if respect_query_parameters was
set, otherwise against path only.

There may be instances with regexp redirects not covering
query parameters as that have not been checked in the past,
thus we need do a second match only against path to keep
them working.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Guido Schmechel +1
  • Verified V +2 Done
    • Benni Mack +2
    • Guido Schmechel +1
    • core-ci +1 CI

Issues 1

Files 4

Review log 5064

Show 3239 earlier messages
  1. Guido Schmechel uploaded patch set 1 06/05/21, 11:58

  2. core-ci voted on patch set 1 V +1 06/05/21, 12:09 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12178

  3. Jochen Roth voted on patch set 1 V +1 CR +1 06/06/21, 08:39

  4. Markus Klein commented on patch set 1 CR -1 07/27/21, 17:50

    Please explain somewhere why we need a distinction between the slash/non-slash variant. IMO you can craft the regex accordingly to accommodate for those scenarios, right?

    RedirectService.php:119

    why error suppression?

    RedirectServiceTest.php:673

    where does the $1 refer to? there is no capture group in the regex
  5. Guido Schmechel uploaded patch set 2 07/28/21, 18:37

  6. Markus Klein commented on patch set 2 07/28/21, 18:48

    Please also answer my other questions. Thank you.

    RedirectServiceTest.php:700

    same here?
  7. core-ci voted on patch set 2 V +1 07/28/21, 18:49 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/13857

  8. Guido Schmechel commented on patch set 2 07/28/21, 19:45

    Reply

    I'm not sure if you have something mixed up here? The regex is taken from the data set and then compared with hits from the "possible Redirects". To be on the safe side, these hits are enriched with a slash. That is why there is a double check here. You can also find this above with the $possibleRedirects in the flat area. (Line 80 - 103)

    RedirectService.php:119 Reply

    I have kept to the previous preg_match here. It was also suppressed there. What is your recommendation?

    RedirectServiceTest.php:673 Reply

    Dumb test, Update coming soon ;-)
  9. Guido Schmechel commented on patch set 2 CR -1 07/28/21, 21:06

    Unfortunately, the new tests are still too stupid. I still have to familiarise myself with the query parameters.
  10. Markus Klein commented on patch set 2 07/28/21, 21:30

    Reply

    Alright, now got the point. I'll probably rephrase the commit message slightly once you're done with coding.

    RedirectService.php:119 Reply

    Good one. I just traced it down to my own patch: https://review.typo3.org/c/Packages/TYPO3.CMS/+/59817/7/typo3/sysext/redirects/Classes/Service/RedirectService.php ;-) It is enough to check for invalid regex once, which happens above. Please remove the @ here.
  11. Guido Schmechel uploaded patch set 3 07/30/21, 20:53

  12. Guido Schmechel uploaded patch set 4 07/30/21, 20:54

  13. core-ci voted on patch set 3 V +1 07/30/21, 21:11 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/13933

  14. core-ci voted on patch set 4 V +1 07/30/21, 21:14 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/13934

  15. Guido Schmechel commented on patch set 4 CR -1 07/30/21, 21:16

    preg_match() for regexp capture groups only works for "$uri->getPath()" RedirectService->replaceRegExpCaptureGroup() must be fixed, too.
  16. Guido Schmechel uploaded patch set 5 07/30/21, 21:52

  17. Guido Schmechel uploaded patch set 6 07/30/21, 21:59

  18. core-ci voted on patch set 6 V +1 07/30/21, 22:09 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/13937

  19. core-ci voted on patch set 5 V +1 07/30/21, 22:11 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/13936

  20. Guido Schmechel commented on patch set 6 07/30/21, 22:17

    What a hassle. Now I know why the tests were such a problem. But now it can be reviewed. Hint1: I have suppressed the first new preg_match because the query is new here compared to the old preg_match. Hint2: The commit message has just been optimised, but can be changed at any time. Hint3: In order for the capture group to work for your own host, a complete domain, whether internal or external, must always be included. This is due to the url resolving: getTargetUrl() in the RedirectService class.
  21. Guido Schmechel commented on patch set 6 07/30/21, 22:17

    RedirectService.php:119 Reply

    Done
  22. Markus Klein commented on patch set 6 08/09/21, 13:04

    RedirectService.php:119

    we don't need to suppress errors here. If the regexp was valid before, it is valid here as well. If it was invalid, we wouldn't get here.

    RedirectService.php:125

    code-wise I would prefer to have this line only once inside the loop. I suggest using a boolean var $isPossibleRedirect, which is set to true, if either of the 3 preg_match returns a positive result
  23. Guido Schmechel uploaded patch set 7 08/14/21, 13:07

  24. core-ci voted on patch set 7 V +1 08/14/21, 13:24 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15076

  25. Guido Schmechel commented on patch set 7 08/14/21, 13:25

    RedirectService.php:119 Reply

    Done

    RedirectService.php:125 Reply

    Done
  26. Markus Klein commented on patch set 7 CR +1 08/16/21, 12:16

    Thanks, nice.
  27. Markus Klein voted on patch set 7 V +1 09/07/21, 11:16

  28. Rudy Gnodde voted on patch set 7 V +1 CR +1 09/08/21, 14:44

  29. Stefan Bürk commented on patch set 7 V -1 CR -1 09/11/21, 23:08

    Firstly thanks for this patch and work, Guido. After thinking deeply about the patch and the way it was done, I had to vote with -1. I hope my comments make clear why and how to mitigate the issues I see.
    Double 'in the' and had a hickup to understand it in the first place. Maybe following would be more understandable: ``` Regular Expression redirects are matched against the path and query, as long request contain the query parameter part. ``` Beside this would like to see more background of the patch in the commit message, as this makes not clear why this is a bugfix in the first place, and why adding respect_query_parameters should be done (option not clear enough in the backend in combination with is_regexp, etc.) Would not have stated it, but as there are more comments/changes needed, leaving it here as comment.
    First, query string is already noted in the sentence above. Secondly the statement with the fragment, the part behind #, is incorrect and missleading. Browser has never send the fragment part to the server, as such only JavaScript or the Browser itself can handle it. Fragment/Hashtag part has been a invention from browser vendors to give the ability to scroll to a defined place in the loaded document. Thus please remove this sentence.

    RedirectService.php:109

    This should be removed, as with changes commented below this get unused.

    RedirectService.php:115

    May you be fine to switch this here to following: ``` if ($matchResult === false) { $this->logger->warning('Invalid regex in redirect', ['regex' => $regexp]); continue; } elseif ($matchResult > 0) { // Regexp already matched, thus continue as further evualtiong this // regexp against path with query would be creating a second possibleRedirect $possibleRedirects += $allRedirects[$domainName]['regexp'][$regexp]; continue; } ```

    RedirectService.php:118

    If above comment is accepted, this check is not needed here anymore, as false matchResults are already skipped.

    RedirectService.php:126

    Instead of setting a flag variable, add the current record as `$possibleRedirect` and `continue;`

    RedirectService.php:132

    This can be removed, as handled with change in comment above.

    RedirectService.php:133

    Not sure with this chained regExp evaulation here at all. So that comment are just some thoughts, maybe you can clarify me if I'm wrong here. Would it not be enough to only preg_match once, either with: - path with query if respect_query_parameters is true - path witout query if respect_query_parameters is false From my point of view, if respect_query_parameter is set, the regexp should handle it if no query params are in the request, or not. Anyway I would go for trim a slash from the path and use this, with eventually the query parameter. Slash and non-slash should be handled by the regExp itself. Do not be confussed because the flat and non-regexp respect_query_parameter checks both variants - first, it is a strict string comparision and second neither the user can know how it comes in, nor the core which the user has added, and as such evualating both variants, which are less costly then multiple preg-matches. For sure this should be documented. As such this whole thing can be reduced to one preg_match call in contrast up to 3 preg_match calls for the same redirect and requested uri.

    RedirectService.php:367

    Adding fragment is unnessesacry. Browsers do not send the fragment (after #) to the server, and as such it cannot be processed on the serverside, neighter with php nor with any other service side language. UriInterface and implementing classes has fragment, as these classes are also used for general uri handling, even more for generation and as such needs this. This method is called to replace the regExpCaptureGroup for the requested uri, and thus adding fragment would never been executed as frament are never send. Please remove it.
  30. Markus Klein commented on patch set 7 09/12/21, 20:57

    RedirectService.php:367 Reply

    Interesting... using Firefox at least the network panel reveals: GET /#foo HTTP/2 Host: reelworx.at if I enter https://reelworx.at/#foo
  31. Stefan Bürk commented on patch set 7 09/12/21, 22:01

    RedirectService.php:367 Reply

    Where in the panel ? Eventuall on top? Or listed in the request header lines, it is not added. Don't let the browser fool you - displaying and sending are different "shoes". You can write a simple php and simple output $_SERVER all other things. Even make a xdebug breakpoint, inspect the request object. You will not find the fragment there. If wanted contact me in slack. You can use following snippet as a index.php ``` <?php echo "<pre>" . PHP_EOL; print_r($_SERVER); print_r($REQUEST); print_r(get_headers()); ``` Or you can create a small golang webserver which simple echos out all it recieves: https://gist.github.com/ppanyukov/f80dd27290e1633641df7c6e0dd727e6 Sry, pure c/c++ prove code was lost, so could not provide that have not found something in short time now. You can search the internet, ansswer are all the same - you can process a string with the hashtag, but you will never recieve it as part of a request. Even webservers directly do not see it (apache, ngnix, .. ). We had requests from customers for this, years ago. We have written even small webserver handerles just to put out what is send. Browsers will never send it. I'm not sure, but ears ago even curl could not send it, maybe it does it now. Or some request libs in differnt languages. But this is against the RFC and unreliable, bug or inteneded for some purpose, but not expecting that servers could handle it. But users / Browser will never send it. You cannot react on it in apache rewrite rules, ngnix stuff, nor in any of the proxy softwares out there. It is simply a pure thing for the client - as long as software builds upon the specification. (Which Typo3 should do too). Furthermore RFC's states this, that a fragment is not part of a URI. It is there, and should be handled by the requesting client after successfull response, and as such not transffered/send with the request: https://www.ietf.org/rfc/rfc2396.txt ``` 4.1. Fragment Identifier When a URI reference is used to perform a retrieval action on the identified resource, the optional fragment identifier, separated from the URI by a crosshatch ("#") character, consists of additional reference information to be interpreted by the user agent after the retrieval action has been successfully completed. As such, it is not part of a URI, but is often used in conjunction with a URI. ``` Here a link to an image, used php snippet above, with opened tool: https://pasteboard.co/UILwM2QON2uQ.png Sry, I would not have stated that so clearly if I wasn't sure, had spend hours to get "this" working. Even talked with some devs. Fragments are absolutly meaningless and usesless for request/response. (Except you can send it as redirection url, because "browsers" executing that redirect, and as such acting as client, are responsible for the hastag. Either it would jump to an anchor link in the document (first and oldest implementation every browser has) - or you nead to handle it with loaded javascript which is loaded with that response. Hope that I could make it a little bit clearer. But feel free to (really) prove you can get the fragment in php from the requested, current uri.
  32. Markus Klein commented on patch set 7 09/13/21, 06:46

    I did not take the time to read all you lengthy comments to full extend, sorry for that. Please feel free to push an updated version of the patch anytime.

    RedirectService.php:367 Reply

    to make things short: The line is not related to this patch at all and is never executed (due to never being present as you clearly stated). So I suggest stopping any lengthy discussion here. Because you asked: FF network panel, raw request headers shows the lines I pasted above. I never said these are the sent ones! I only found it to interesting to know that there is a discrepancy.
  33. Guido Schmechel commented on patch set 7 09/13/21, 08:34

    Thank you for your accurate review. I will take care of an update in a few days :-)
  34. Stefan Bürk commented on patch set 7 09/13/21, 08:40

    Reply

    Hi Guido, I just prepared a new patchset with my noted changes. Besides this came up to have one case further in there with a comment, otherwise it would be breaking. Just need/want to check and add more tests to it, have the feeling that there are some missing. So if you are fine with it I would finish this after work and push a new patchset; please let me know if you do not want it, and want it to do it on your own.
  35. Guido Schmechel commented on patch set 7 09/13/21, 08:43

    Reply

    All fine. Otherwise it would be double work ;-) Feel free to push :-)
  36. Stefan Bürk uploaded patch set 8 09/13/21, 09:58

  37. Stefan Bürk voted on patch set 8 CR -1 09/13/21, 09:58

  38. Stefan Bürk commented on patch set 8 09/13/21, 09:59

    Just pushed polishing for now. Still needs some unit, and even more functional tests. Will take care of it after work.
  39. Stefan Bürk commented on patch set 8 09/13/21, 10:00

    RedirectService.php:367 Reply

    Patch is
  1. Stefan Bürk commented on patch set 8 09/13/21, 10:02

    Reply

    Done

    Reply

    Done

    RedirectService.php:109 Reply

    Done

    RedirectService.php:115 Reply

    Done

    RedirectService.php:118 Reply

    Done

    RedirectService.php:126 Reply

    Done

    RedirectService.php:132 Reply

    Done

    RedirectService.php:133 Reply

    Done
  2. core-ci voted on patch set 8 V +1 09/13/21, 10:08 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16638

  3. Stefan Bürk rebased onto patch set 9 09/20/21, 21:10

  4. core-ci voted on patch set 9 V +1 09/20/21, 21:25 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16955

  5. Stefan Bürk uploaded patch set 10 10/03/21, 16:35

  6. Stefan Bürk commented on patch set 10 10/03/21, 16:37

    Reply

    Done
    Added functional tests.

    RedirectServiceTest.php:673 Reply

    Done

    RedirectServiceTest.php:700 Reply

    Done
  7. core-ci voted on patch set 10 V +1 10/03/21, 16:48 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17514

  8. Stefan Bürk voted on patch set 10 V +1 CR +1 10/03/21, 17:47

  9. Stefan Bürk wrote 10/03/21, 19:02

    Removed Code-Review+1 by Stefan Bürk <stefan@buerk.tech>
  10. Stefan Bürk wrote 10/03/21, 19:02

    Removed Verified+1 by Stefan Bürk <stefan@buerk.tech>
  11. Stefan Bürk commented on patch set 10 CR -1 10/03/21, 19:29

    RedirectService.php:382

    In short: If respect_query_parameters is true, and with example regexp group eventually query parameters will be added as part of matching group. Needs further investigation.
  12. Stefan Bürk uploaded patch set 11 10/04/21, 03:26

  13. core-ci voted on patch set 11 V -1 10/04/21, 03:44 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17526

  14. Stefan Bürk uploaded patch set 12 10/04/21, 04:08

  15. core-ci voted on patch set 12 V -1 10/04/21, 04:28 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17527

  16. Stefan Bürk uploaded patch set 13 10/04/21, 04:37

  17. core-ci voted on patch set 13 V +1 10/04/21, 04:58 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17528

  18. Benni Mack updated the commit message in patch set 14 10/04/21, 09:15

  19. core-ci voted on patch set 14 V +1 10/04/21, 09:31 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/17540

  20. Stefan Bürk commented on patch set 14 10/04/21, 10:05

    RedirectService.php:382 Reply

    Done
  21. Stefan Bürk commented on patch set 14 10/04/21, 10:06

    RedirectService.php:382 Reply

    Covered by tests and clean up is in there. (Only forgot to note it)
  22. Benni Mack voted on patch set 14 V +1 CR +1 10/04/21, 15:14

  23. Guido Schmechel voted on patch set 14 V +1 CR +1 10/04/21, 15:41

  24. Benni Mack voted on patch set 14 V +2 CR +2 10/04/21, 16:47

  25. Benni Mack merged patch set 15 10/04/21, 16:47