Portia Spider logs showing ['Partial'] during crawling

谁说胖子不能爱 提交于 2019-12-01 01:06:44

Late answer, but hopefully not useless, since this behavior by scrapy doesn't seem well-documented. Looking at this line of code from the scrapy source, the partial flag is set when the request encounters a Twisted PotentialDataLoss error. According to the corresponding Twisted documentation:

This only occurs when making requests to HTTP servers which do not set Content-Length or a Transfer-Encoding in the response

Possible causes include:

  • The server is misconfigured
  • There's a proxy involved that's blocking some headers
  • You get a response that doesn't normally have Content-Length, e.g. redirects (301, 302, 303), but you've set handle_httpstatus_list or handle_httpstatus_all such that the response doesn't get filtered out by HttpErrorMiddleware or fetched by RedirectMiddleware
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!