Is waiting for the page part of the crawl delay?

后端 未结 0 821
北恋
北恋 2021-01-30 09:18

I\'ve been undertaking some fairly large scraping projects (>10000 pages), and some of the websites I am getting information from ask for a crawl delay of 10 seconds. However

相关标签:
回答
  • 消灭零回复
提交回复
热议问题