Reliable non blocking reads from subprocess stdout

问题

Note: I have a process that writes one line to stdout ("print("hello")) and waits for raw_input. I run this process with p = subprocess.Popen, then call p.stdout.readline()....this blocks indefinitely. I am setting shell=False...Why can I not read that first output?

p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin = subprocess.PIPE)
    print(p.stdout.readline())

I have seen this thread about non blocking io for subprocess. I am trying to create an interactive subprocess object that reads and writes to the process. I found that stdout is blocking, even though there is output sent to stdout. To test this, I wrote a program that spits increasing integers to stdout and called this via subprocess. I then used the non blocking method involving a Queue to read stdout. What I found was that, despite the fact that the process is spitting 1000 lines to stdout per second, Stdout blocks for long periods at a time, returning a line completely randomly...one in every thousand, then one in 10 etc. Any thoughts on why this might be happening? If I slow down the printer to print once every 0.5 seconds, I never get a read from stdout. Its like it needs 1000 writes to sdtout before it responds with a single line.

Note: I put a sleep in so there was time between starting the listen thread and reading from it. Now, I always return from stdout, but I get a random single line. I guess I don't understand stdout. Does it not buffer? I mean, I want to get everything that is spit out by my process, but it seems like it only saves the last line. I can't find documentation on these PIPE things.

  def getStdOut(self):
        '''
            reads standard out
        '''
        return self.nonBlockingRead()


    def enqueue_output(self, out, queue, kill):
        line = ""
        kill = True
        while kill:
            line = out.readline()
            queue.put(line)

#             for line in iter(out.readline, b''):
#                 queue.put(line)
#             out.close()

    def nonBlockingRead(self):
        '''
             taken from the internet
        '''
        import sys
        from subprocess import PIPE, Popen
        from threading  import Thread
        sleep(0.5) #I inserted this later
        try:
            from Queue import Queue, Empty
        except ImportError:
            from queue import Queue, Empty  # python 3.x

        ON_POSIX = 'posix' in sys.builtin_module_names

        killThread = False                
        self.q = Queue()
        t = Thread(target=self.enqueue_output, args=(self.theProcess.stdout, self.q, killThread))
        t.daemon = True # thread dies with the program
        t.start()

        line = ''
        try:  
            line = self.q.get_nowait() # or q.get(timeout=.1)
        except Exception as e:
            print(e)
            print('no output yet')
        killThread = True
        return line




if __name__ == "__main__" :
    '''
        unit tests
    '''
    import time
    cmd = 'python "C:\InteractiveTest.py"'
    aSubProc = subProcessWStdIn(cmd, 10)
    while True :    
#        print(aSubProc.writeToProcess('y'))
        print(aSubProc.getStdOut())

回答1:

readline reads a single line from the file-like object PIPE, to read it all of it, simply wrap it in a while loop. You should also call sleep after each read to save on CPU cycles.

Here is a simple example:

import subprocess

p = subprocess.Popen(
    ['ls', '-lat'],
    shell=False,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    stdin=subprocess.PIPE
)
while True:
    line = p.stdout.readline()
    if line == '':
        break
    print(line.strip())  # remove extra ws between lines

EDIT:

woah, sorry, I completely missed the part you were trying to read input in that other process...

So, fn your other process, looks something like:

print('Hello')
in = raw_input()

Then the print actually sends the content to the file-like object PIPE you passed earlier which has it's own buffering mechanism. This behavior is explained in the print() function docs

To solve this simply add a sys.stdout.flush() between your print and raw_input:

print('Hello')
sys.stdout.flush()  # "flush" the output to our PIPE
in = raw_input()

来源：https://stackoverflow.com/questions/22044253/reliable-non-blocking-reads-from-subprocess-stdout

标签

python

subprocess

stdout

nonblocking