How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out

Question

I want output from execute Test_Pipe.py, I tried following code on Linux but it did not work.

Test_Pipe.py

import time
while True :         


        

Answer 1

Nadia's snippet does work but calling read with a 1 byte buffer is highly unrecommended. The better way to do this would be to set the stdout file descriptor to nonblocking using fcntl

fcntl.fcntl(
    proc.stdout.fileno(),
    fcntl.F_SETFL,
    fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL) | os.O_NONBLOCK,
)

and then using select to test if the data is ready

while proc.poll() == None:
    readx = select.select([proc.stdout.fileno()], [], [])[0]
    if readx:
        chunk = proc.stdout.read()
        print chunk

She was correct in that your problem must be different from what you posted as Caller.py and Test_Pipe.py do work as provided.

• https://derrickpetzold.com/p/capturing-output-from-ffmpeg-python/

Answer 2

Test_Pipe.py buffers its stdout by default so proc in Caller.py doesn't see any output until the child's buffer is full (if the buffer size is 8KB then it takes around a minute to fill Test_Pipe.py's stdout buffer).

To make the output unbuffered (line-buffered for text streams) you could pass -u flag to the child Python script. It allows to read subprocess' output line by line in "real-time":

import sys
from subprocess import Popen, PIPE

proc = Popen([sys.executable, "-u", "Test_Pipe.py"], stdout=PIPE, bufsize=1)
for line in iter(proc.stdout.readline, b''):
    print line,
proc.communicate()

See links in Python: read streaming input from subprocess.communicate() on how to solve the block-buffering issue for non-Python child processes.

Answer 3

You obviously can use subprocess.communicate but I think you are looking for real time input and output.

readline was blocked because the process is probably waiting on your input. You can read character by character to overcome this like the following:

import subprocess
import sys

process = subprocess.Popen(
    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

while True:
    out = process.stdout.read(1)
    if out == '' and process.poll() != None:
        break
    if out != '':
        sys.stdout.write(out)
        sys.stdout.flush()

Answer 4

To avoid the many problems that can always arise with buffering for tasks such as "getting the subprocess's output to the main process in real time", I always recommend using pexpect for all non-Windows platform, wexpect on Windows, instead of subprocess, when such tasks are desired.