- scrapy环境的安装
a. pip3 install wheel
b. 下载twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
c. 进入下载目录，执行 pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl
d. pip3 install pywin32
e. pip3 install scrapy
- scrapy使用
- 1.创建一个工程：scrapy startproject ProName （注 startproject 中间没有空格）
- 2.cd ProName
- 3.创建爬虫文件：scrapy genspider first www.xxx.com
- 4.执行：（allowed_domains 注释掉）
- settings.py:
- 不遵从rbotes协议
- 进行UA伪装
- 指定日志等级：LOG_LEVEL = ‘ERROR’
scrapy crawl spiderName
- 持久化存储
- 基于终端指令：
- 前提：只可以将parse方法的返回值进行本地文件的持久化存储
- 指令：scrapy crawl spiderName -o filePath
- 基于管道：
- 编码流程：
1.数据解析
2.需要在item类中定义相关的属性（存储解析到的数据）(在 items.py 中添加属性)
3.将解析到的数据存储或者封装到一个item类型的对象中（在 主文件里 实例化item 封装数据） （注是item['name'] 不是item.name）
4.将item对象提交到管道中(yield item)
5.在管道中需要接收item，且将item对象中的数据进行任意形式的持久化操作（在piplines.py里添加
）
6.在配置文件中开启管道（settings 里打开 ITEM_PIPELINES）5.scrapy如何实现持久化存储？
- 1.数据解析
- 2.封装item类（定义属性）
- 3.将解析的数据存储到item对象
- 4.将item提交到管道
- 5.在管道中接受item且对其进行任意i形式的持久化存储
- 6.在配置文件中开启管道
6.简述基于移动端数据爬取的配置流程？
- 下载fiddler
- 配置fiddler：
- 配置端口
- allow remote xxx
- 在手机端下载安装证书
- 手机和fiddler的pc端处于同一个网段（pc分享一个热点手机连接）
- 在手机浏览器中访问ip：port下载证书
- 在手机中设置网络代理：代理信息都是来源于fiddler所在的pc
scrapy管道的细节处理
- 数据的爬取和持久化存储，将同一组数据分别存储到不同的平台
- 一个管道类负责将数据存储到某一个载体或者平台中
- 爬虫文件提交的item只会提交给第一个被执行的管道类
- 在管道类的process_item中的return item表示的含义就是将当前管道类接收到的item传递给下一个即将被执行的管道类
- 注意：爬虫类和管道类进行数据交互的形式
- yild item：只可以交互item类型的对象
- spider参数：可以交互任意形式的数据
- 全站数据的爬取
- 基于手动请求发送实现
- 实现全站数据爬取
- 实现深度爬取
- 手动请求发送：
- yield scrapy.Request(url,callback)
- yield scrapy.FormRequest(url,formdata,callback)
- post请求的发送和cookie的处理
- cookie的处理是scrapy自动封装好
- start_urls自动请求发送的实现原理：
#是父类的一个请求发送的封装
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url=url,callback=self.parse)
- 深度爬取
- 手动请求发送
- 请求传参：持久化存储,将不同的回调中解析的数据存储到同一个item对象。请求传参传递的就是item对象。
- 使用场景：如果使用scrapy爬取的数据没有在同一张页面中
- 传递方式：将传递数据封装到meta字典中，meta传递个了callback
yield scrapy.Request(url,callback,meta)
- 接收：
在callback指定的回调函数中使用response进行接收：
- item = response.meta['key']
- 五大核心组件
- 引擎(Scrapy)
用来处理整个系统的数据流处理, 触发事务(框架核心)
- 调度器(Scheduler)
用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址
- 下载器(Downloader)
用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的)
- 爬虫(Spiders)
爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面
- 项目管道(Pipeline)
负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。
- 如何提升scrapy爬取数据的效率
增加并发：
默认scrapy开启的并发线程为32个，可以适当进行增加。在settings配置文件中修改CONCURRENT_REQUESTS = 100值为100,并发设置成了为100。
降低日志级别：
在运行scrapy时，会有大量日志信息的输出，为了减少CPU的使用率。可以设置log输出信息为INFO或者ERROR即可。在配置文件中编写：LOG_LEVEL = ‘INFO’
禁止cookie：
如果不是真的需要cookie，则在scrapy爬取数据时可以禁止cookie从而减少CPU的使用率，提升爬取效率。在配置文件中编写：COOKIES_ENABLED = False
禁止重试：
对失败的HTTP进行重新请求（重试）会减慢爬取速度，因此可以禁止重试。在配置文件中编写：RETRY_ENABLED = False
减少下载超时：
如果对一个非常慢的链接进行爬取，减少下载超时可以能让卡住的链接快速被放弃，从而提升效率。在配置文件中进行编写：DOWNLOAD_TIMEOUT = 10 超时时间为10s
BOSS直聘深度爬取
import scrapy
from bossDeepPro.items import BossdeepproItem
class BossSpider(scrapy.Spider):
name = 'boss'
# allowed_domains = ['www.xxx.com']
start_urls = ['https://www.zhipin.com/job_detail/?query=python%E5%BC%80%E5%8F%91&city=101010100&industry=&position=']
# 通用的url模板（不可变）
url = 'https://www.zhipin.com/c101010100/?query=python开发&page=%d'
page = 2
def parse(self, response):
print('正在爬取第{}页的数据'.format(self.page))
li_list = response.xpath('//*[@id="main"]/div/div[3]/ul/li | //*[@id="main"]/div/div[2]/ul/li')
for li in li_list:
job_title = li.xpath('.//div[@class="info-primary"]/h3/a/div[1]/text()').extract_first()
salary = li.xpath('.//div[@class="info-primary"]/h3/a/span/text()').extract_first()
#实例化item对象：对象必须要让parse和parse_detail共享
item = BossdeepproItem()
item['job_title'] = job_title
item['salary'] = salary
detail_url = 'https://www.zhipin.com'+li.xpath('.//div[@class="info-primary"]/h3/a/@href').extract_first()
#对详情页的url发起手动请求
yield scrapy.Request(url=detail_url,callback=self.parse_detail,meta={'item':item}) #(多个管道间的数据传输)
if self.page <= 5:
# 对其他页码进行手动请求的发送
new_url = format(self.url % self.page)
print(new_url)
self.page += 1
# 手动请求发送
# callback进行数据解析
yield scrapy.Request(url=new_url, callback=self.parse)
#解析岗位描述
def parse_detail(self,response):
item = response.meta['item']
job_desc = response.xpath('//*[@id="main"]/div[3]/div/div[2]/div[2]/div[1]/div//text()').extract()
job_desc = ''.join(job_desc)
item['job_desc'] = job_desc
yield itemimport scrapy
class BossdeepproItem(scrapy.Item):
# define the fields for your item here like:
job_title = scrapy.Field()
job_desc = scrapy.Field()
salary = scrapy.Field()# -*- coding: utf-8 -*-
# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
from scrapy import signals
class BossproSpiderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the spider middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_spider_input(self, response, spider):
# Called for each response that goes through the spider
# middleware and into the spider.
# Should return None or raise an exception.
return None
def process_spider_output(self, response, result, spider):
# Called with the results returned from the Spider, after
# it has processed the response.
# Must return an iterable of Request, dict or Item objects.
for i in result:
yield i
def process_spider_exception(self, response, exception, spider):
# Called when a spider or process_spider_input() method
# (from other spider middleware) raises an exception.
# Should return either None or an iterable of Request, dict
# or Item objects.
pass
def process_start_requests(self, start_requests, spider):
# Called with the start requests of the spider, and works
# similarly to the process_spider_output() method, except
# that it doesn’t have a response associated.
# Must return only requests (not items).
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
class BossproDownloaderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the downloader middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_request(self, request, spider):
# Called for each request that goes through the downloader
# middleware.
# Must either:
# - return None: continue processing this request
# - or return a Response object
# - or return a Request object
# - or raise IgnoreRequest: process_exception() methods of
# installed downloader middleware will be called
return None
def process_response(self, request, response, spider):
# Called with the response returned from the downloader.
# Must either;
# - return a Response object
# - return a Request object
# - or raise IgnoreRequest
return response
def process_exception(self, request, exception, spider):
# Called when a download handler or a process_request()
# (from other downloader middleware) raises an exception.
# Must either:
# - return None: continue processing this exception
# - return a Response object: stops process_exception() chain
# - return a Request object: stops process_exception() chain
pass
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)class BossdeepproPipeline(object):
def process_item(self, item, spider):
print(item)
return item# -*- coding: utf-8 -*-
# Scrapy settings for bossDeepPro project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'bossDeepPro'
SPIDER_MODULES = ['bossDeepPro.spiders']
NEWSPIDER_MODULE = 'bossDeepPro.spiders'
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'bossDeepPro (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
LOG_LEVEL = 'ERROR'
# LOG_FILE = 'log.txt'
# Configure maximum concurrent requests performed by Scrapy (default: 16)
# CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'bossDeepPro.middlewares.BossdeepproSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'bossDeepPro.middlewares.BossdeepproDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'bossDeepPro.pipelines.BossdeepproPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
来源：oschina
链接：https://my.oschina.net/u/4275499/blog/3437694