Scrapy - Correct way to change User Agent in Request
问题 I have created a custom Middleware in Scrapy by overriding the RetryMiddleware which changes both Proxy and User-Agent before retrying. It looks like this class CustomRetryMiddleware(RetryMiddleware): def _retry(self, request, reason, spider): retries = request.meta.get('retry_times', 0) + 1 if retries <= self.max_retry_times: Proxy_UA_Middleware.switch_proxy() Proxy_UA_Middleware.switch_ua() logger.debug("Retrying %(request)s (failed %(retries)d times): %(reason)s", {'request': request,