Robots协议一定放在网站根目录下

生来就可爱ヽ(ⅴ<●) 提交于 2019-12-04 15:44:40

一.网络爬虫的尺寸

1.以爬取网页,玩转网页为目的进行小规模,数据量小对爬取速度不敏感的可以使用request库实现功能(占90%)

2.以爬取网站或爬取系列网站为目的,比如说获取一个或多个旅游网站的爬虫,对数据要求规模较大,爬取速度敏感的可以使用Scrapy库

3.以爬取全网为目的,规模很大搜索引擎爬取速度关键,需要定制开发

二.网络爬虫带来的问题

总的来说有:骚扰问题,法律风险,隐私泄露

1.爬虫可利用计算机的快速功能访问服务器,它会比人类的速度快到百倍甚至千倍,受限于编写水平和目的,网络爬虫将会为web服务器带来巨大的资源开销。对网站运行者来讲,爬虫形成了骚扰。

2.网络爬虫会带来法律风险。服务器上的数据有产权归属,比如新浪上的新闻规新浪所有,如果网络爬虫获取数据牟利后将会带来法律风险。

3.网络爬虫会造成隐私泄露。网络爬虫可能具备突破简单访问控制的能力,或得被保护数据从而泄露个人隐私。   

三.网络爬虫的限制

来源审查:判断User-Agent进行限制

检查来访HTTP协议头的User-Agent域,只响应浏览器或友好爬虫的访问。

发布公告: Robots协议

告知所有爬虫网站的爬取策略,要求爬虫遵守。

四.Robots协议

作用:网站会告知网络爬虫哪些页面可以抓取,哪些不行

形式:在网站根目录下的robots.txt文件。

下面我自己做了一个实例  打开京东的Robots协议 https://www.jd.com/robots.txt

会出现下面几句话

 

 大概的意思是所有爬虫都要遵守以下协定

任何爬虫都不允许访问?* 即?号开头的网页

任何爬虫都不允许访问pop*.html开头的网页

任何爬虫都不允许访问pinpai*.html开头的网页

此外 EtaoSpider HuihuiSpidder GwdangSpider WochachaSpider这四个网络爬虫都不能爬取京东的任何数据

五.Robots协议基本语法

*代表所有          /代表根目录

User-agent:  代表哪些爬虫

Disallow:你不允许访问资源的目录

 

六.其他注意

1.Robots协议一定放在网站根目录下

2.http://www.news.sina.com.cn/robots.txt和http://www.sina.com.cn/robots.txt它们的Robots协议是不一样的

 3.如果一个网站没有Robots协议,是允许所有爬虫无限制爬取

http://market.szonline.net/amaz/24026.html
http://market.szonline.net/amaz/24025.html
http://market.szonline.net/amaz/24024.html
http://market.szonline.net/amaz/24023.html
http://market.szonline.net/amaz/24022.html
http://market.szonline.net/amaz/24021.html
http://market.szonline.net/amaz/24020.html
http://market.szonline.net/amaz/24019.html
http://market.szonline.net/amaz/24018.html
http://market.szonline.net/amaz/24017.html
http://market.szonline.net/amaz/24016.html
http://market.szonline.net/amaz/24015.html
http://market.szonline.net/amaz/24014.html
http://market.szonline.net/amaz/24013.html
http://market.szonline.net/amaz/24012.html
http://market.szonline.net/amaz/24011.html
http://market.szonline.net/amaz/24010.html
http://market.szonline.net/amaz/24009.html
http://market.szonline.net/amaz/24008.html
http://market.szonline.net/amaz/24007.html
http://market.szonline.net/amaz/24040.html
http://market.szonline.net/amaz/24039.html
http://market.szonline.net/amaz/24038.html
http://market.szonline.net/amaz/24037.html
http://market.szonline.net/amaz/24102.html
http://market.szonline.net/amaz/24101.html
http://market.szonline.net/amaz/24100.html
http://market.szonline.net/amaz/24099.html
http://market.szonline.net/amaz/24098.html
http://market.szonline.net/amaz/24097.html
http://market.szonline.net/amaz/24096.html
http://market.szonline.net/amaz/24095.html
http://market.szonline.net/amaz/24094.html
http://market.szonline.net/amaz/24093.html
http://market.szonline.net/amaz/24092.html
http://market.szonline.net/amaz/24091.html
http://market.szonline.net/amaz/24090.html
http://market.szonline.net/amaz/24089.html
http://market.szonline.net/amaz/24088.html
http://market.szonline.net/amaz/24087.html
http://market.szonline.net/amaz/24086.html
http://www.wu0553.com/news/58292.html
http://www.wu0553.com/news/58291.html
http://www.wu0553.com/news/58289.html
http://www.wu0553.com/news/58288.html
http://www.wu0553.com/news/58286.html
http://www.wu0553.com/news/58284.html
http://www.wu0553.com/news/58283.html
http://www.wu0553.com/news/58281.html
http://www.wu0553.com/news/58280.html
http://www.wu0553.com/news/58278.html
http://www.wu0553.com/news/58276.html
http://www.wu0553.com/news/58273.html
http://www.wu0553.com/news/58271.html
http://www.wu0553.com/news/58270.html
http://www.wu0553.com/news/58268.html
http://www.wu0553.com/news/58266.html
http://www.wu0553.com/news/58265.html
http://www.wu0553.com/news/58263.html
http://www.wu0553.com/news/58262.html
http://www.wu0553.com/m/view.php?aid=58292
http://www.wu0553.com/m/view.php?aid=58291
http://www.wu0553.com/m/view.php?aid=58289
http://www.wu0553.com/m/view.php?aid=58288
http://www.wu0553.com/m/view.php?aid=58286
http://www.wu0553.com/m/view.php?aid=58284
http://www.wu0553.com/m/view.php?aid=58283
http://www.wu0553.com/m/view.php?aid=58281
http://www.wu0553.com/m/view.php?aid=58280
http://www.wu0553.com/m/view.php?aid=58278
http://www.wu0553.com/m/view.php?aid=58276
http://www.wu0553.com/m/view.php?aid=58273
http://www.wu0553.com/m/view.php?aid=58271
http://www.wu0553.com/m/view.php?aid=58270
http://www.wu0553.com/m/view.php?aid=58268
http://www.wu0553.com/m/view.php?aid=58266
http://www.wu0553.com/m/view.php?aid=58265
http://www.wu0553.com/m/view.php?aid=58263
http://www.wu0553.com/m/view.php?aid=58262
http://www.wu0553.com/m/view.php?aid=58260
http://www.wu0553.com/m/view.php?aid=58258
http://www.wu0553.com/m/view.php?aid=58257
http://www.wu0553.com/m/view.php?aid=58255
http://www.wu0553.com/m/view.php?aid=58254
http://www.wu0553.com/m/view.php?aid=58252
http://www.wu0553.com/m/view.php?aid=58250
http://www.wu0553.com/m/view.php?aid=58249
http://www.wu0553.com/m/view.php?aid=58247
http://www.wu0553.com/m/view.php?aid=58242
http://www.wu0553.com/m/view.php?aid=58218
http://www.wu0553.com/m/view.php?aid=58214
http://www.wu0553.com/m/view.php?aid=58213
http://www.wu0553.com/news/58260.html
http://www.wu0553.com/news/58258.html
http://www.wu0553.com/news/58257.html
http://www.wu0553.com/news/58255.html
http://www.wu0553.com/news/58254.html
http://www.wu0553.com/news/58252.html
http://www.wu0553.com/news/58250.html
http://www.wu0553.com/news/58249.html
http://www.wu0553.com/news/58247.html
http://www.wu0553.com/news/58242.html
http://www.wu0553.com/news/58218.html
http://www.wu0553.com/news/58214.html
http://www.wu0553.com/news/58213.html
http://market.szonline.net/amaz/24053.html
http://market.szonline.net/amaz/24052.html
http://market.szonline.net/amaz/24051.html
http://market.szonline.net/amaz/24050.html
http://market.szonline.net/amaz/24049.html
http://market.szonline.net/amaz/24048.html
http://market.szonline.net/amaz/24047.html
http://market.szonline.net/amaz/24046.html
http://market.szonline.net/amaz/24045.html
http://market.szonline.net/amaz/24044.html
http://market.szonline.net/amaz/24043.html
http://market.szonline.net/amaz/24042.html
http://market.szonline.net/amaz/24041.html
http://market.szonline.net/amaz/24036.html
http://market.szonline.net/amaz/24035.html
http://market.szonline.net/amaz/24034.html
http://market.szonline.net/amaz/24033.html
http://market.szonline.net/amaz/24032.html
http://market.szonline.net/amaz/24031.html
http://market.szonline.net/amaz/24030.html
http://market.szonline.net/amaz/24029.html
http://market.szonline.net/amaz/24028.html
http://market.szonline.net/amaz/24027.html

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!