Scrapy: how to catch download error and try download it again

Question

During my crawling, some pages failed due to unexpected redirection and no response returned. How can I catch this kind of error and re-schedule a request with original url, not

Answer 1

you can override the RETRY_HTTP_CODES in settings.py

This is the settings I use for proxy errors:

RETRY_HTTP_CODES = [500, 502, 503, 504, 400, 403, 404, 408]