how to get the original start_url in scrapy (before redirect)

Question

I\'m using Scrapy to crawl some pages. I fetch the start_urls from an excel sheet and I need to save the url in the item.

class abc_Spider(BaseSpider):
   name =         


        

Answer 1

You can find what you need in response.request.meta['redirect_urls'].

Quote from docs:

The urls which the request goes through (while being redirected) can be found in the redirect_urls Request.meta key.

Hope that helps.