问题
I want to add nofollow and noindex to my site whilst it's being built. The client has request I use these rules.
I am aware of
<meta name="robots" content="noindex,nofollow">
But I only have access to the robots.txt file.
Does anyone know the correct format I can use to apply noindex, nofollow rules via the robots.txt file?
回答1:
noindex and nofollow means you do not want your site to crawl in search engine.
so simply put code in robots.txt
User-agent: *
Disallow: /
it means noindex and nofollow.
回答2:
There is a non-standard Noindex field, which Google (and likely no other consumer) supported as experimental feature.
Following the robots.txt specification, you can’t disallow indexing nor following links with robots.txt.
For a site that is still in development, has not been indexed yet, and doesn’t get backlinks from pages which may be crawled, using robots.txt should be sufficient:
# no bot may crawl
User-agent: *
Disallow: /
If pages from the site are already indexed, and/or if other pages which may be crawled link to it, you have to use noindex, which can not only be specified in the HTML, but also as HTTP header:
X-Robots-Tag: noindex, nofollow
回答3:
- Noindex tells search engines not to include pages in search results, but can follow links (and also can transfer PA and DA)
- Nofollow tells bots not to follow the links. We also can combine noindex with follow in pages we don´t want to be indexed, but we want to follow the links
来源：https://stackoverflow.com/questions/45790024/how-to-add-nofollow-noindex-all-pages-in-robots-txt