hadoop streaming: how to see application logs?

Question

I can see all hadoop logs on my /usr/local/hadoop/logs path

but where can I see application level logs? for example :

mapper.py

Answer 1

Hadoop streaming uses STDIN/STDOUT for passing the key/value pairs between the mappers and reducers, so the log messages have to be written to a specific log file - check the sample code and the python logging documentation for more details. This Query might also help.

Answer 2

Hadoop gathers stderr, which can be viewed on hadoop map/reduce status site. So you can just write to stderr.