A non-blocking read on a subprocess.PIPE in Python

后端 未结 29 2556
醉酒成梦
醉酒成梦 2020-11-21 04:49

I\'m using the subprocess module to start a subprocess and connect to its output stream (standard output). I want to be able to execute non-blocking reads on its standard ou

相关标签:
29条回答
  • 2020-11-21 05:07

    In my case I needed a logging module that catches the output from the background applications and augments it(adding time-stamps, colors, etc.).

    I ended up with a background thread that does the actual I/O. Following code is only for POSIX platforms. I stripped non-essential parts.

    If someone is going to use this beast for long runs consider managing open descriptors. In my case it was not a big problem.

    # -*- python -*-
    import fcntl
    import threading
    import sys, os, errno
    import subprocess
    
    class Logger(threading.Thread):
        def __init__(self, *modules):
            threading.Thread.__init__(self)
            try:
                from select import epoll, EPOLLIN
                self.__poll = epoll()
                self.__evt = EPOLLIN
                self.__to = -1
            except:
                from select import poll, POLLIN
                print 'epoll is not available'
                self.__poll = poll()
                self.__evt = POLLIN
                self.__to = 100
            self.__fds = {}
            self.daemon = True
            self.start()
    
        def run(self):
            while True:
                events = self.__poll.poll(self.__to)
                for fd, ev in events:
                    if (ev&self.__evt) != self.__evt:
                        continue
                    try:
                        self.__fds[fd].run()
                    except Exception, e:
                        print e
    
        def add(self, fd, log):
            assert not self.__fds.has_key(fd)
            self.__fds[fd] = log
            self.__poll.register(fd, self.__evt)
    
    class log:
        logger = Logger()
    
        def __init__(self, name):
            self.__name = name
            self.__piped = False
    
        def fileno(self):
            if self.__piped:
                return self.write
            self.read, self.write = os.pipe()
            fl = fcntl.fcntl(self.read, fcntl.F_GETFL)
            fcntl.fcntl(self.read, fcntl.F_SETFL, fl | os.O_NONBLOCK)
            self.fdRead = os.fdopen(self.read)
            self.logger.add(self.read, self)
            self.__piped = True
            return self.write
    
        def __run(self, line):
            self.chat(line, nl=False)
    
        def run(self):
            while True:
                try: line = self.fdRead.readline()
                except IOError, exc:
                    if exc.errno == errno.EAGAIN:
                        return
                    raise
                self.__run(line)
    
        def chat(self, line, nl=True):
            if nl: nl = '\n'
            else: nl = ''
            sys.stdout.write('[%s] %s%s' % (self.__name, line, nl))
    
    def system(command, param=[], cwd=None, env=None, input=None, output=None):
        args = [command] + param
        p = subprocess.Popen(args, cwd=cwd, stdout=output, stderr=output, stdin=input, env=env, bufsize=0)
        p.wait()
    
    ls = log('ls')
    ls.chat('go')
    system("ls", ['-l', '/'], output=ls)
    
    date = log('date')
    date.chat('go')
    system("date", output=date)
    
    0 讨论(0)
  • 2020-11-21 05:08

    I have created a library based on J. F. Sebastian's solution. You can use it.

    https://github.com/cenkalti/what

    0 讨论(0)
  • 2020-11-21 05:08

    Try wexpect, which is the windows alternative of pexpect.

    import wexpect
    
    p = wexpect.spawn('myprogram.exe')
    p.stdout.readline('.')               // regex pattern of any character
    output_str = p.after()
    
    0 讨论(0)
  • 2020-11-21 05:09

    The select module helps you determine where the next useful input is.

    However, you're almost always happier with separate threads. One does a blocking read the stdin, another does wherever it is you don't want blocked.

    0 讨论(0)
  • 2020-11-21 05:13

    I have the original questioner's problem, but did not wish to invoke threads. I mixed Jesse's solution with a direct read() from the pipe, and my own buffer-handler for line reads (however, my sub-process - ping - always wrote full lines < a system page size). I avoid busy-waiting by only reading in a gobject-registered io watch. These days I usually run code within a gobject MainLoop to avoid threads.

    def set_up_ping(ip, w):
    # run the sub-process
    # watch the resultant pipe
    p = subprocess.Popen(['/bin/ping', ip], stdout=subprocess.PIPE)
    # make stdout a non-blocking file
    fl = fcntl.fcntl(p.stdout, fcntl.F_GETFL)
    fcntl.fcntl(p.stdout, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    stdout_gid = gobject.io_add_watch(p.stdout, gobject.IO_IN, w)
    return stdout_gid # for shutting down
    

    The watcher is

    def watch(f, *other):
    print 'reading',f.read()
    return True
    

    And the main program sets up a ping and then calls gobject mail loop.

    def main():
    set_up_ping('192.168.1.8', watch)
    # discard gid as unused here
    gobject.MainLoop().run()
    

    Any other work is attached to callbacks in gobject.

    0 讨论(0)
  • 2020-11-21 05:13

    I also faced the problem described by Jesse and solved it by using "select" as Bradley, Andy and others did but in a blocking mode to avoid a busy loop. It uses a dummy Pipe as a fake stdin. The select blocks and wait for either stdin or the pipe to be ready. When a key is pressed stdin unblocks the select and the key value can be retrieved with read(1). When a different thread writes to the pipe then the pipe unblocks the select and it can be taken as an indication that the need for stdin is over. Here is some reference code:

    import sys
    import os
    from select import select
    
    # -------------------------------------------------------------------------    
    # Set the pipe (fake stdin) to simulate a final key stroke
    # which will unblock the select statement
    readEnd, writeEnd = os.pipe()
    readFile = os.fdopen(readEnd)
    writeFile = os.fdopen(writeEnd, "w")
    
    # -------------------------------------------------------------------------
    def getKey():
    
        # Wait for stdin or pipe (fake stdin) to be ready
        dr,dw,de = select([sys.__stdin__, readFile], [], [])
    
        # If stdin is the one ready then read it and return value
        if sys.__stdin__ in dr:
            return sys.__stdin__.read(1)   # For Windows use ----> getch() from module msvcrt
    
        # Must finish
        else:
            return None
    
    # -------------------------------------------------------------------------
    def breakStdinRead():
        writeFile.write(' ')
        writeFile.flush()
    
    # -------------------------------------------------------------------------
    # MAIN CODE
    
    # Get key stroke
    key = getKey()
    
    # Keyboard input
    if key:
        # ... do your stuff with the key value
    
    # Faked keystroke
    else:
        # ... use of stdin finished
    
    # -------------------------------------------------------------------------
    # OTHER THREAD CODE
    
    breakStdinRead()
    
    0 讨论(0)
提交回复
热议问题