Order of items in a very basic robots.txt changing the command scope?

前端 未结 0 410
孤街浪徒
孤街浪徒 2020-11-21 21:18

I had a very simple robots.txt file setup for a site I maintain. After a spike of traffic that the ISP put down to crawlers they suggested I add a crawl delay directive wh

相关标签:
回答
  • 消灭零回复
提交回复
热议问题