How to save Scrapy crawl Command output

Question

I am trying to save the output of the scrapy crawl command I have tried scrapy crawl someSpider -o some.json -t json >> some.text But it doesn\'t worked ...c

Answer 1

You need to redirect stderr too. You are redirecting only stdout. You can redirect it somehow like this:

scrapy crawl someSpider -o some.json -t json 2> some.text

The key is number 2, which "selects" stderr as source for redirection.

If you would like to redirect both stderr and stdout into one file, you can use:

scrapy crawl someSpider -o some.json -t json &> some.text

For more about output redirection: http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html