Nohup and Python -u : it still doesn't log data in realtime

Question

When launching a Python process, in background, with

nohup python myscript.py > test.log 2>&1 < /dev/null &

the problem is

Answer 1

If python -u does not seem to be working for you, the next suggestion would be to replace sys.stdout with a custom class that does not buffer output.

Magnus Lycka provided this solution in a mailing list

class Unbuffered(object):
    def __init__(self, stream):
        self.stream = stream
    def write(self, data):
        self.stream.write(data)
        self.stream.flush()
    def __getattr__(self, attr):
        return getattr(self.stream, attr)

import sys
sys.stdout = Unbuffered(sys.stdout)
print 'Unbuffered Output'

I can't see why python -u wouldn't work from your example, but this should solve the issue for you.

Answer 2

There are multiple ways of handling this.

1. Create your own un-buffered output function and use it instead of print()

import sys

def log(msg):
    sys.stdout.write(msg)
    sys.stdout.flush()

log("Hello World!")

2. Use mkfifo command to create your own logging device and write to it instead of stdout. Use a separate command to pipe from device to a file.

3. Write a log file directly and flush when you do. Same as number one, but avoids using stdout altogether. This is what I would do.

I would be suspicious of nohup. Terminal opens standard i/o streams and passes them to processes. If you run a process with nohup and then logout and close the terminal, I am not sure what happens to the standard streams. It might be that OS does some sort of garbage collect and closes them leaving your process without stdout. You should really consider writing your own logs.