“500 Internal Server Error” when combining Scrapy over Splash with an HTTP proxy

Question

I\'m trying to crawl a Scrapy spider in a Docker container using both Splash (to render JavaScript) and Tor through Privoxy (to provide anonymity). Here is the docker-

Answer 1

Following the structure of the Aquarium project as suggested by paul trmbrth, I found that it is essential to name the .ini file default.ini, not proxy.ini (otherwise it doesn't get 'picked up' automatically). I managed to get the scraper to work in this way (cf. my self-answer to How to use Scrapy with both Splash and Tor over Privoxy in Docker Compose).