Saving items from Scrapyd to Amazon S3 using Feed Exporter

Question

Using Scrapy with amazon S3 is fairly simple, you set:

• FEED_URI = \'s3://MYBUCKET/feeds/%(name)s/%(time)s.jl\'
• FEED_FORMAT = \'jsonlines\'
• A

Answer 1

I also faced the same problem. Removing the items_dir= from scrapyd.conf file worked for me.

Answer 2

You can set the items_dir property to an empty value like this:

[scrapyd]
items_dir=

It seems that when that property is set, takes precedence over the configured exported. See http://scrapyd.readthedocs.org/en/latest/config.html for more information.