How to decrease the bandwidth of scraping pages via Scrapy

若如初见. 提交于 2020-01-26 03:58:04

问题


I am using Scrapy with Luminati proxy to scrape thousands of Amazon pages, but I noticed my scraping bandwidth consumption is very high. I am scraping the whole page right now and I am just wondering if it is possible to remove/block images, css, js because I am just dealing with html code to make the scraping bandwidth as minimum as possible.

thank you for looking into my problem :)

来源:https://stackoverflow.com/questions/59541755/how-to-decrease-the-bandwidth-of-scraping-pages-via-scrapy

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!