subprocess.Popen communicate() writes to console, but not to log file

问题

I have the following line in a Python script that runs a separate Python script from within the original script:

subprocess.Popen("'/MyExternalPythonScript.py' " + theArgumentToPassToPythonScript, shell=True).communicate()

Using the above line, any print() statements found in the separate Python file do appear in the console of the main Python script.

However, these statements are not reflected in the .txt file log that the script writes to.

Does anyone know how to fix this, so that the .txt file exactly reflects the true console text of the main Python script?

This is the method I am using to save the console as a .txt file, in real time:

import sys
class Logger(object):
    def __init__(self):
        self.terminal = sys.stdout
        self.log = open("/ScriptLog.txt", "w", 0)
    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)


sys.stdout = Logger()

I am not necessarily attached to this method. I am interested in any method that will achieve what I've detailed.

回答1:

Keep in mind that subprocess spawns a new process, and doesn't really communicate with the parent process (they're pretty much independent entities). Despite its name, the communicate method is just a way of sending/receiving data from the parent process to the child process (simulate that the user input something on the terminal, for instance)

In order to know where to write the output, subprocess uses numbers (file identifiers or file numbers). When subprocess spawns a process, the child process only knows that the standard output is the file identified in the O.S. as 7 (to say a number) but that's pretty much it. The subprocess will independently query the operative system with something like "Hey! What is file number 7? Give it to me, I have something to write in it." (understanding what a C fork does is quite helpful here)

Basically, the spawned subprocess doesn't understand your Logger class. It just knows it has to write its stuff to a file: a file which is uniquely identified within the O.S with a number and that unless otherwise specified, that number corresponds with the file descriptor of the standard output (but as explained in the case #2 below, you can change it if you want)

So you have several "solutions"...

Clone (tee) stdout to a file, so when something is written to stdout, the operative system ALSO writes it to your file (this is really not Python-related... it's OS related):

import os
import tempfile
import subprocess

file_log = os.path.join(tempfile.gettempdir(), 'foo.txt')
p = subprocess.Popen("python ./run_something.py | tee %s" % file_log, shell=True)
p.wait()

Choose whether to write to terminal OR to the file using the fileno() function of each. For instance, to write only to the file:

import os
import tempfile
import subprocess

file_log = os.path.join(tempfile.gettempdir(), 'foo.txt')
with open(file_log, 'w') as f:
    p = subprocess.Popen("python ./run_something.py", shell=True, stdout=f.fileno())
    p.wait()

What I personally find "safer" (I don't feel confortable overwriting sys.stdout): Just let the command run and store its output into a variable and pick it up later (in the parent process):

import os
import tempfile
import subprocess

p = subprocess.Popen("python ./run_something.py", shell=True, stdout=subprocess.PIPE)
p.wait()
contents = p.stdout.read()
# Whatever the output of Subprocess was is now stored in 'contents'
# Let's write it to file:
file_log = os.path.join(tempfile.gettempdir(), 'foo.txt')
with open(file_log, 'w') as f:
    f.write(contents)

This way, you can also do a print(contents) somewhere in your code to output whatever the subprocess "said" to the terminal.

For example purposes, the script "./run_something.py" is just this:

print("Foo1")
print("Foo2")
print("Foo3")

回答2:

Do you really need subprocess.Popen's communicate() method? It looks like you just want the output. That's what subprocess.check_output() is for.

If you use that, you can use the built-in logging module for "tee"-ing the output stream to multiple destinations.

import logging
import subprocess
import sys

EXTERNAL_SCRIPT_PATH = '/path/to/talker.py'
LOG_FILE_PATH = '/path/to/debug.log'

logger = logging.getLogger('')
logger.setLevel(logging.INFO)

# Log to screen
console_logger = logging.StreamHandler(sys.stdout)
logger.addHandler(console_logger)

# Log to file
file_logger = logging.FileHandler(LOG_FILE_PATH)
logger.addHandler(file_logger)

# Driver script output
logger.info('Calling external script')

# External script output
logger.info(
    subprocess.check_output(EXTERNAL_SCRIPT_PATH, shell=True)
)

# More driver script output
logger.info('Finished calling external script')

As always, be careful with shell=True. If you can write the call as subprocess.check_output(['/path/to/script.py', 'arg1', 'arg2']), do so!

来源：https://stackoverflow.com/questions/47999670/subprocess-popen-communicate-writes-to-console-but-not-to-log-file

标签

python

subprocess