The robots.txt file is one of the ways search engines use to determine what pages they can crawl on your website. All the top search engines support robots.txt.
⚠️ Important: Don't go any further without advice from your SEO professionals. Otherwise you might do more harm than good.
Location of your robots.txt file
By default you will have a robots.txt file on your site found at
https://yourDomain.com/robots.txt
By default this file has the recommended defaults to encourage the search engines to crawl your website.
👍 ProTip: Your website is SEO friendly and encourages Google and other search engines to crawl your website. You don't need to do anything, that happens by default.
Uploading a new file
If your SEO company provides you with a new custom robots.txt file, then you will need to compare it with the default file and then only add the new items that aren't included in the default. The store merges the default robots with your custom uploaded robots file.
⚠️ Important: Only add the new additions that you need included in the file. The store will automatically merge your file with the existing default robots.txt file.
⚠️ Important: This file is case sensitive, so make sure your upload the correct txt file with all lowercase "robots.txt"
Upload to the Media Manager
Upload the "robots.txt" to the root of your media manager (e.g. not inside of any folder).
That's it now check it's working by visiting https://yourDomain.com/robots.txt
Submit to Google
Google Robots Tester Tool
Click to "see live robots.txt" and review it is correct
Then click submit and choose to submit to Google
Dangerzone setting
You might be wondering about the "Dangerzone" setting and what relation it has to your robots.txt. The meta tags are controlled by the dangerzone setting.
The search engine will first look at your robots.txt file and then determine what pages can be crawled. Then when it lands on a page it will read the meta tags set by the dangerzone toggle.
Thus, the Meta Tags and the Robots file are slightly different but related. One of them doesn't override the other one.
These can come in conflict which is why you should probably just stick to the defaults, or ask your SEO professionals.
Well done! Learn more about SEO here.