Review main Abandoned

60738[FEATURE] Instruct search engines to only index pages when in production context

Georg Ringer CR 0 V 0 Normal +110 −0 updated 02/15/20, 09:58

Commit message

To avoid indexing non production content the header `X-Robots-Tag` is set
to 'none' if the Application Context is not set to `Production` or any
sub-context.

Also the robots meta tag is set to `noindex,nofollow` to make sure search
engines don't get confused.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 6

Review log 5266

Show 3441 earlier messages
  1. Georg Ringer uploaded patch set 1 05/13/19, 08:31

  2. Georg Ringer commented on patch set 1 05/13/19, 08:31

    This change is ready for review.
  3. TYPO3com voted on patch set 1 V +1 05/13/19, 08:44 Completed build in 13m 07s on Mon, 13 May, 10:44 AM Test Summary: 47686 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-33840

  4. Josef Glatz commented on patch set 1 05/13/19, 09:38

    I like the approach! At this point I really would like to see the possibility, configuring it also for environments like Production/Staging, Production/Dev, ... (e.g. via TYPO3_CONF_VARS OR ENV var? or another more modern approach)
  5. Oliver Klee voted on patch set 1 CR +1 05/13/19, 09:39

  6. Guido Schmechel commented on patch set 1 05/13/19, 09:57

    > Patch Set 1: > > I like the approach! At this point I really would like to see the possibility, configuring it also for environments like Production/Staging, Production/Dev, ... (e.g. via TYPO3_CONF_VARS OR ENV var? or another more modern approach) Maybe we can revert the condition like !isProduction()? So we don't need another configuration possibillity.
  7. Georg Ringer commented on patch set 1 05/13/19, 10:55

    > Maybe we can revert the condition like !isProduction()? So we don't need another configuration possibillity. Production/Fo is also Production and there is only Development or Production. Testing is for unit/functional tests and not for actual sites
  8. Richard Haeser commented on patch set 1 05/13/19, 11:06

    Let me start by saying that this is definitely a feature that needs to find it's way to core. On the other hand, I have to think about if this is the right way by using the header. We already set the meta tag with robot instructions and that may "confuse" Google because those values can differ now. Besides that: you only fix the problem of indexing and to be sure, you even would like Google to not crawl your website at all.
  9. Fabien Udriot voted on patch set 1 CR +1 05/13/19, 12:11

  10. Frank Nägler voted on patch set 1 CR +1 05/13/19, 19:13

  11. Frank Nägler voted on patch set 1 V +1 05/13/19, 19:24

  12. Richard Haeser commented on patch set 1 V -1 05/13/19, 19:25

    A -1 to prevent it to get merged before I had the time to check this.
  13. Richard Haeser commented on patch set 1 CR -2 05/13/19, 19:42

    I’m checking the http header in combination with the meta tag with our friends of Yoast to see if they have done research on this.
  14. Richard Haeser commented on patch set 1 05/13/19, 20:11

    Ok, got feedback. And as I already expected: or the header tag and the meta tag should have the same value, or it is best to only use the meta tag if that one is already there. BTW: best way to prevent indexing your site is by far using basic auth or ip-whitelisting to only accept those people who needs access, but as you are all professionals you all know that. To prevent indexing by accident, this is a nice feature!
  15. Josef Glatz commented on patch set 1 05/14/19, 06:07

    > Patch Set 1: > > Ok, got feedback. And as I already expected: or the header tag and the meta tag should have the same value, or it is best to only use the meta tag if that one is already there. > > BTW: best way to prevent indexing your site is by far using basic auth or ip-whitelisting to only accept those people who needs access, but as you are all professionals you all know that. To prevent indexing by accident, this is a nice feature! When I get the need for the patch correctly it's intended trying to prevent indexing automatically. (from a generic POV) -> That's exactly that what I do in my projects for years via vhost configuration or .htaccess. The patch adds the header to every kind of request (of course no files), which can't be done via meta tag. This is really helpful. For my actual 9.5 projects I use a condition in TypoScript to check if it's a non-productive TYPO3 instance by my own (https://github.com/josefglatz/TYPO3-Distribution/blob/8160d9b607e750aea7ab69138fe8fc00ccffb9fe/app/web/typo3conf/ext/theme/Configuration/TypoScript/Base/Page/Head.setup.typoscript#L122 ff) and overwrite the `page.meta.robots` tag. @Richard, if that can be accomplished automatically it would be very nice. So what about extending the patch in a way, that the robots meta tag is also set automatically? --- > I really would like to know what are the actions of others here to protect a staging instance `Production/Staging` or `Production/PlaygroundForCustomer` from Google? (BESIDE using Basic Auth or IP limitations) My experiences until now: some of our customers doesn't want a Basic Authentication or IP limitation. Especially when they want to test their upcoming website on a broader audience (mobile phones, mobile phones in the wild, mobile phones with 3G).

    DisallowSearchEngineIndexResponseHeader.php:44 Unresolved

    What do you think about extending this if() to make it possible for configuring it via TYPO3_CONF_VARS? I often have Environments with production context running in subcontexts like Show, Staging, Dev which is a playground for the customer or just a test/preview system for the develop branch of a project (where all relevant team members can test it before it get's merged into master).
  16. Kevin Appelt commented on patch set 1 CR -1 05/22/19, 04:04

    Nice feature but I would prefer: 1) having the possibility to set the related contexts by myself. (Like adding Production/Staging) 2) that the meta tag changes too to avoid misunderstandings. 3) having the possibility to disable this behaviour (for whatever reason) - but this is not so important for me
  17. Josef Glatz voted on patch set 1 CR -1 05/22/19, 09:16

  18. Richard Haeser commented on patch set 1 05/22/19, 18:07

    I would really like to have the same output for the meta tag. We can do that! Personally I'm totally fine with only opening up for search engines when in Production context. But what about having the possibility to manually set it in the Site Configuration as well? So by default, it will set the header and add no index robot meta tags to all sites that are not in production context, unless you specify it within your Site management (including overrides).
  19. Richard Haeser commented on patch set 1 09/02/19, 06:24

    As this is a feature we can definitely use in core, let's get this patch forward. Will work on a new patchset.
  20. Richard Haeser uploaded patch set 2 09/06/19, 10:10

  21. TYPO3com voted on patch set 2 V +1 09/06/19, 10:20 Completed build in 09m 00s on Fri, 6 Sep, 10:20 AM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36122

  22. Richard Haeser uploaded patch set 3 09/06/19, 10:43

  23. Richard Haeser voted on patch set 3 CR 0 09/06/19, 10:43

  24. TYPO3com voted on patch set 3 V +1 09/06/19, 10:56 Completed build in 12m 51s on Fri, 6 Sep, 10:56 AM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36125

  25. Richard Haeser uploaded patch set 4 09/06/19, 11:05

  26. TYPO3com voted on patch set 4 V +1 09/06/19, 11:17 Completed build in 12m 04s on Fri, 6 Sep, 11:17 AM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36127

  27. Benjamin Franzke uploaded patch set 5 09/06/19, 12:17

  28. Frank Nägler commented on patch set 5 09/06/19, 12:21

    first time I saw this patch I thought this is a good idea. But now I think this behavior should not be bind to the context but e.g. to configuration setting. I guess there could be a reason to run a production site in development context, yes I know bad practice, and I am not a fan of this. but if you bind this behavior to the context, a site in development context will never be indexed. This is a very hard limitation. anyway, if we decide to do this, the rst file contains invalid information I guess.

    Feature-88339-X-Robots-TagHeaderIfApplicationContextIsSetToDevelopment.rst:12 Unresolved

    or *not* any sub-context... (I guess)

    Feature-88339-X-Robots-TagHeaderIfApplicationContextIsSetToDevelopment.rst:22 Unresolved

    *not* in production context?
  29. Frank Nägler voted on patch set 5 CR -1 09/06/19, 12:22

  30. TYPO3com voted on patch set 5 V +1 09/06/19, 12:29 Completed build in 11m 30s on Fri, 6 Sep, 12:29 PM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36139

  31. Frank Nägler commented on patch set 5 09/06/19, 12:36

    oh, and it is a kind of breaking feature
  32. Richard Haeser commented on patch set 5 09/06/19, 12:38

    > Patch Set 5: > > (2 comments) > > first time I saw this patch I thought this is a good idea. > But now I think this behavior should not be bind to the context but e.g. to configuration setting. I guess there could be a reason to run a production site in development context, yes I know bad practice, and I am not a fan of this. but if you bind this behavior to the context, a site in development context will never be indexed. > This is a very hard limitation. > > anyway, if we decide to do this, the rst file contains invalid information I guess. I'm in favor of having an option to disable this behaviour, but in my opinion it should work out of the box when you're configuring your site the right way. Hopefully we will help people with things like this to configure it the "right" way as well. Personally I can't think of a reason why you would like to have a website publicly available without setting it to the production context. I also moved the middleware to the SEO extension, so it is only used when you want to have SEO enhancements. Will think about the right way how to disable the feature if you really don't want this to happen.
  33. Frank Nägler commented on patch set 5 CR 0 09/06/19, 12:42

    > I also moved the middleware to the SEO extension, so it is only used when you want to have SEO enhancements. you say the right word "middleware", for the case I have in mind, I am able to disable the middleware. So there is an option to disable the behavior and I am fine. Please check the remarks in the rst, then I am fine to get this in.
  34. Richard Haeser commented on patch set 5 09/06/19, 13:07

    > Patch Set 5: -Code-Review > > > I also moved the middleware to the SEO extension, so it is only used when you want to have SEO enhancements. > > you say the right word "middleware", for the case I have in mind, I am able to disable the middleware. So there is an option to disable the behavior and I am fine. > Please check the remarks in the rst, then I am fine to get this in. Maybe a good idea, to mention this in the rst?
  35. Richard Haeser commented on patch set 5 09/06/19, 13:09

    > Patch Set 5: > > oh, and it is a kind of breaking feature Hmm... Only breaking if you did not configure your site correctly. So it that really breaking?
  36. Andreas Kienast voted on patch set 5 CR +1 09/06/19, 13:24

  37. Richard Haeser uploaded patch set 6 09/06/19, 13:37

  38. Andreas Kienast uploaded patch set 7 09/06/19, 13:44

  39. TYPO3com voted on patch set 6 V +1 09/06/19, 13:48 Completed build in 10m 25s on Fri, 6 Sep, 01:48 PM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36143

  40. TYPO3com voted on patch set 7 V +1 09/06/19, 13:54 Completed build in 10m 16s on Fri, 6 Sep, 01:54 PM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36144

  41. Andreas Kienast voted on patch set 7 V +1 CR +1 09/06/19, 13:55

  1. Andreas Kienast commented on patch set 7 V 0 CR -1 09/06/19, 14:13

    Some things I found: - in your ReST you mention the middleware can be disabled. How do I do that with the code in MetaTagGenerator - in the middleware, you set X-Robots-Tag to "noindex" only
  2. Richard Haeser commented on patch set 7 09/06/19, 14:16

    On it
  3. Richard Haeser uploaded patch set 8 09/06/19, 14:51

  4. Richard Haeser commented on patch set 8 09/06/19, 14:56

    DefaultConfiguration.php:78 Unresolved

    Not sure if we want to put this feature in EXT:core while EXT:seo where it is used, is optional.
  5. TYPO3com voted on patch set 8 V +1 09/06/19, 15:00 Completed build in 08m 58s on Fri, 6 Sep, 03:00 PM Test Summary: 47369 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36146

  6. Benjamin Franzke commented on patch set 8 CR -1 09/06/19, 15:18

    DisallowSearchEngineIndexResponseHeader.php:49 Unresolved

    Please inject Features as constructor dependency.

    DisallowSearchEngineIndexResponseHeader.php:53 Unresolved

    A gree with andy, that we'd need to inlude nofollow here. I think we should actually use 'none' here, which is a shortcut for 'noindex, nofollow' – according to google: https://developers.google.com/search/reference/robots_meta_tag#valid-indexing--serving-directives Would save some bytes..
  7. Richard Haeser uploaded patch set 9 09/06/19, 15:35

  8. TYPO3com voted on patch set 9 V -1 09/06/19, 15:45 Completed build in 09m 27s on Fri, 6 Sep, 03:45 PM Test Summary: 1 of 47737 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36148

  9. TYPO3com voted on patch set 9 V +1 09/06/19, 15:47 Completed build in 20m 27s on Fri, 6 Sep, 03:47 PM Test Summary: 47378 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36148

  10. TYPO3com voted on patch set 9 V -1 09/06/19, 15:56 Completed build in 08m 58s on Fri, 6 Sep, 03:56 PM Test Summary: 47131 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36149

  11. TYPO3com voted on patch set 9 V +1 09/06/19, 16:00 Completed build in 22m 05s on Fri, 6 Sep, 04:00 PM Test Summary: 47378 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36149

  12. Helmut Hummel commented on patch set 9 09/10/19, 09:55

    MetaTagGenerator.php:119

    Please use Environment::getContext()
  13. Helmut Hummel commented on patch set 9 09/10/19, 09:58

    DisallowSearchEngineIndexResponseHeader.php:45

    Does DI work properly here? In terms of: is a new object created or is DI configured to use the global one?
  14. Richard Haeser uploaded patch set 10 09/11/19, 18:53

  15. Richard Haeser commented on patch set 10 09/11/19, 19:02

    MetaTagGenerator.php:119 Reply

    Done
  16. TYPO3com voted on patch set 10 V +1 09/11/19, 19:03 Completed build in 10m 04s on Wed, 11 Sep, 07:03 PM Test Summary: 47515 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36329

  17. Benjamin Franzke commented on patch set 10 09/11/19, 19:07

    DisallowSearchEngineIndexResponseHeader.php:45 Reply Unresolved

    Yes, the early ApplicationContext instance – as created and added to the container by Bootstrap::init() – will be injected here.
  18. Susi Moog commented on patch set 10 09/12/19, 05:47

    Feature-88339-X-Robots-TagHeaderIfApplicationContextIsSetToDevelopment.rst:14

    this is now "none", isn't it?
  19. Richard Haeser uploaded patch set 11 09/13/19, 10:47

  20. Richard Haeser commented on patch set 11 09/13/19, 10:56

    Feature-88339-X-Robots-TagHeaderIfApplicationContextIsSetToDevelopment.rst:14 Reply

    Done
  21. TYPO3com voted on patch set 11 V +1 09/13/19, 11:08 Completed build in 19m 39s on Fri, 13 Sep, 11:08 AM Test Summary: 47536 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36414

  22. Markus Poerschke commented on patch set 11 CR -1 09/18/19, 07:38

    This change will disallow search engines to index a TYPO3 page that are not configured with TYPO3_CONTEXT "Production". I see some problems with this patch: 1. Stage-Systems are deployed using the TYPO3_CONTEXT "Production". Therefore this change will not prevent those systems beeing indexed, if no HTTP Basic Auth is in place. The "Production" context is important to test functionality like caching. 2. If the TYPO3 system is deployed with no TYPO3_CONTEXT set explicitly, then the "Production" context will be used as default. This means that stage systems that are accidentally accessible to the public are using the "Production" context. Therefore I think it is needed to accomplish these change in a different manner. For example the indexation could be disabled by: 1. TypoScript constant. 2. Site configuration. 3. By using a regular expression, so that only the primary host name of a site is indexable. 4. Only rely on the feature toogle which needs to be activated manually only on the stage systems. I see benefits in implementing 3.) because this will also prevent doublicate content on production site if the sites are available under different hosts.
  23. Frank Nägler commented on patch set 11 CR -1 09/20/19, 13:52

    I agree here with Markus Poerschke. Binding this behavior to the TYPO3 context is not a good idea. I opt for a setting e.g. in TYPO3_CONF_VARS which can be set by an environment variable or something else.
  24. Susi Moog commented on patch set 11 01/25/20, 11:59

    ping @Georg: will you take care of the last comments and change your implementation accordingly?
  25. Benni Mack abandoned the change 02/15/20, 09:58

    Abandoned for now as discussed with Georg and Richard