robots.txt with no user-agent specified?

前端 未结 0 1748
旧时难觅i
旧时难觅i 2021-01-29 01:20

A website I want to scrape has this robots.txt:

User-agent: Disallow: /wp-login.php
Disallow: /wp-admin/
Crawl-delay: 10

How do I interpret this

相关标签:
回答
  • 消灭零回复
提交回复
热议问题