Issue with scraping JS rendered page with Scrapy and Splash

Question

I\'m trying to scrape this page which includes following html according to chrome

            Orange Paired

        
         


        

Answer 1

Good starting point in such cases is to look at FAQ section of Splash documentation. It turns out that in your case you need to disable Private mode for Splash, either via --disable-private-mode startup option for Docker, or by setting splash.private_mode_enabled = false in your LUA script.

Once you disable Private mode, the page renders correctly.