I am currently writing a web crawler using Scrapy on Python. I will be scraping about 4,000,000 pages every day, resulting in about 10 values (floats) scraped per page, per