I\'m using scrapy to crawl an e-commerce website I\'m experienced with simpler websites where scrapy alone or with splash/selenium handle most cases. I have a new situation wher