Rails robots.txt folders

Question

I\'m about to launch a Rails app and as the last task, I wan\'t to set the robots.txt file. I couldn\'t find information about how the paths should be written properly for a Rai

Answer 1

you have to put your robots.txt in /public folder

and it will look like in below

The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/" or "/tmp/", or /foo.html:

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

for more information please follow below links

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

and

http://www.robotstxt.org/robotstxt.html