How to save Scrapy crawl Command output

Question

I am trying to save the output of the scrapy crawl command I have tried scrapy crawl someSpider -o some.json -t json >> some.text But it doesn\'t worked ...c

Answer 1

You need to redirect stderr too. You are redirecting only stdout. You can redirect it somehow like this:

scrapy crawl someSpider -o some.json -t json 2> some.text

The key is number 2, which "selects" stderr as source for redirection.

If you would like to redirect both stderr and stdout into one file, you can use:

scrapy crawl someSpider -o some.json -t json &> some.text

For more about output redirection: http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html

Answer 2

For all scrapy commands you can add --logfile NAME_OF_FILE to log to a file e.g.

scrapy crawl someSpider -o some.json --logfile some.text

There are two other useful command line options for logging:

• -L or --loglevel to control the logging level e.g. -L INFO (the default is DEBUG)

• --nolog to disable logging completely

These commands are documented here.

Answer 3

you can use nohup:

nohup scrapy crawl someSpider &

The log will be stored in nohup.out

Answer 4

You can add these lines to your settings.py:

LOG_STDOUT = True
LOG_FILE = '/tmp/scrapy_output.txt'

And then start your crawl normally:

scrapy crawl someSpider

Answer 5

You can save as log file:

scrapy crawl someSpider -s LOG_FILE=fileName.log -L <loglevel>

loglevel can be one between CRITICAL, ERROR, WARNING, INFO and DEBUG or --nolog for no log. for more information read Doc.

Answer 6

scrapy crawl someSpider --logfile some.text

This will do exactly what you are looking for, save the output of the command that you see on the screen to a text file.