Robots.txt is store-view on admin panel, but global on disk

[remkoj-ism]

While testing the SEO features of M2, I've identified a second issue, in addition to the bug described at #1146 at least the generated robots.txt is not compatible with a multi-site setup

Steps to reproduce

1. Setup Magento 2 with two store views, each on their own domain.
2. Go to the design settings (Stores > Configuration > General > Design)
3. Ensure that you're in global scope
4. Under Search Engine Robots load the default robots.txt by pressing "Reset to Default"
5. Save configuration
6. Check 1: /robots.txt (should be in /pub when using /pub as web-root, see A few issues when you use /pub as DocumentRoot #1146) contains the just saved configuration.
7. Switch to one of the store-views
8. Replace the content in "Edit custom instruction of robots.txt File" with different content
9. Save configuration
10. Check 2: /robots.txt (should be in /pub when using /pub as web-root, see A few issues when you use /pub as DocumentRoot #1146) contains the just saved configuration. There's still one file with the content of the store-view, the content of the global scope just got lost.

Suggested solution

I see three options:

• Make the robots.txt settings global, so they match the implemented logic
• Hide the robots.txt section in production mode
• Do not write the robots.txt to disk, but provide a controller which will render the robots.txt based on the configuration and active store-view. Leverage caching by Varnish to prevent M2 from rendering the robots.txt often.

[mcspronko]

Hi Magento team,

Any updates regarding this issue?

Thanks,
Max

[hostep]

I really like option 3 of @remkoj-ism's suggestions

It would allow extension developers / integrators to also add logic to add contents to the robots.txt file.

For example: I'm thinking about 3rd party layered navigation modules, which can sometimes add a whole bunch of query parameters to the product listing url's depending on how you filter/sort/search for products. In most of the cases, you don't want a search engine to index all those different combination of filters/sorting/search params.

If those extensions were able to manipulate the contents of the robots.txt through some kind of API, they could dynamically add custom rules to the robots.txt about certain query parameters which should be ignored by search engines.

Even Magento 2's stock layered navigation can benefit from this I believe.

The key to this is caching, we don't want every request to run a whole bunch of queries against the database to built this file obviously.

This is just a wild idea, might turn out to actually be a bad idea, I haven't really thought it through a lot :)

[duhon]

Hello @remkoj-ism

Please, provide the used version. If the problem is actual for a specific tag, please, specify it and be sure that the latest update was used.

[mcspronko]

Hi Andrii ( @duhon ),

This issue exists for all Magento 2.x versions.
Please have a look into \Magento\Config\Model\Config\Backend\Admin\Robots class
$this->_directory = $filesystem->getDirectoryWrite(DirectoryList::ROOT);

Hope it helps to resolve the issue faster since it is a blocker for all production websites.

Thanks

[mcspronko]

Any progress on this issue? At least internal ticket number :).
Thanks

[mcspronko]

Any updates regarding this issue? It affects all websites and its SEO/Marketing goals.

[hostep]

@mcspronko, over at #1146 they mention MAGETWO-47607, so I guess this ticket also gets tracked in that issue.

[pboisvert]

yes--Piotr expanded the scope of 47607 to include robots.txt in addition to sitemap

[magento-engcom-team]

@remkoj-ism, thank you for your report.
The issue is already fixed in develop branch, 2.2.0