Running Scrapy from a script with file output

Question

I\'m currently using Scrapy with the following command line arguments:

scrapy crawl my_spider -o data.json

However, I\'d prefer to \'save\' thi

Answer 1

You need to add the FEED_FORMAT and FEED_URI to your CrawlerProcess:

process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'FEED_FORMAT': 'json',
'FEED_URI': 'data.json'
})