A non-blocking read on a subprocess.PIPE in Python

后端 未结 29 2555
醉酒成梦
醉酒成梦 2020-11-21 04:49

I\'m using the subprocess module to start a subprocess and connect to its output stream (standard output). I want to be able to execute non-blocking reads on its standard ou

相关标签:
29条回答
  • 2020-11-21 05:14

    why bothering thread&queue? unlike readline(), BufferedReader.read1() wont block waiting for \r\n, it returns ASAP if there is any output coming in.

    #!/usr/bin/python
    from subprocess import Popen, PIPE, STDOUT
    import io
    
    def __main__():
        try:
            p = Popen( ["ping", "-n", "3", "127.0.0.1"], stdin=PIPE, stdout=PIPE, stderr=STDOUT )
        except: print("Popen failed"); quit()
        sout = io.open(p.stdout.fileno(), 'rb', closefd=False)
        while True:
            buf = sout.read1(1024)
            if len(buf) == 0: break
            print buf,
    
    if __name__ == '__main__':
        __main__()
    
    0 讨论(0)
  • 2020-11-21 05:15

    This solution uses the select module to "read any available data" from an IO stream. This function blocks initially until data is available, but then reads only the data that is available and doesn't block further.

    Given the fact that it uses the select module, this only works on Unix.

    The code is fully PEP8-compliant.

    import select
    
    
    def read_available(input_stream, max_bytes=None):
        """
        Blocks until any data is available, then all available data is then read and returned.
        This function returns an empty string when end of stream is reached.
    
        Args:
            input_stream: The stream to read from.
            max_bytes (int|None): The maximum number of bytes to read. This function may return fewer bytes than this.
    
        Returns:
            str
        """
        # Prepare local variables
        input_streams = [input_stream]
        empty_list = []
        read_buffer = ""
    
        # Initially block for input using 'select'
        if len(select.select(input_streams, empty_list, empty_list)[0]) > 0:
    
            # Poll read-readiness using 'select'
            def select_func():
                return len(select.select(input_streams, empty_list, empty_list, 0)[0]) > 0
    
            # Create while function based on parameters
            if max_bytes is not None:
                def while_func():
                    return (len(read_buffer) < max_bytes) and select_func()
            else:
                while_func = select_func
    
            while True:
                # Read single byte at a time
                read_data = input_stream.read(1)
                if len(read_data) == 0:
                    # End of stream
                    break
                # Append byte to string buffer
                read_buffer += read_data
                # Check if more data is available
                if not while_func():
                    break
    
        # Return read buffer
        return read_buffer
    
    0 讨论(0)
  • 2020-11-21 05:16

    Python 3.4 introduces new provisional API for asynchronous IO -- asyncio module.

    The approach is similar to twisted-based answer by @Bryan Ward -- define a protocol and its methods are called as soon as data is ready:

    #!/usr/bin/env python3
    import asyncio
    import os
    
    class SubprocessProtocol(asyncio.SubprocessProtocol):
        def pipe_data_received(self, fd, data):
            if fd == 1: # got stdout data (bytes)
                print(data)
    
        def connection_lost(self, exc):
            loop.stop() # end loop.run_forever()
    
    if os.name == 'nt':
        loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows
        asyncio.set_event_loop(loop)
    else:
        loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(loop.subprocess_exec(SubprocessProtocol, 
            "myprogram.exe", "arg1", "arg2"))
        loop.run_forever()
    finally:
        loop.close()
    

    See "Subprocess" in the docs.

    There is a high-level interface asyncio.create_subprocess_exec() that returns Process objects that allows to read a line asynchroniosly using StreamReader.readline() coroutine (with async/await Python 3.5+ syntax):

    #!/usr/bin/env python3.5
    import asyncio
    import locale
    import sys
    from asyncio.subprocess import PIPE
    from contextlib import closing
    
    async def readline_and_kill(*args):
        # start child process
        process = await asyncio.create_subprocess_exec(*args, stdout=PIPE)
    
        # read line (sequence of bytes ending with b'\n') asynchronously
        async for line in process.stdout:
            print("got line:", line.decode(locale.getpreferredencoding(False)))
            break
        process.kill()
        return await process.wait() # wait for the child process to exit
    
    
    if sys.platform == "win32":
        loop = asyncio.ProactorEventLoop()
        asyncio.set_event_loop(loop)
    else:
        loop = asyncio.get_event_loop()
    
    with closing(loop):
        sys.exit(loop.run_until_complete(readline_and_kill(
            "myprogram.exe", "arg1", "arg2")))
    

    readline_and_kill() performs the following tasks:

    • start subprocess, redirect its stdout to a pipe
    • read a line from subprocess' stdout asynchronously
    • kill subprocess
    • wait for it to exit

    Each step could be limited by timeout seconds if necessary.

    0 讨论(0)
  • 2020-11-21 05:16

    One solution is to make another process to perform your read of the process, or make a thread of the process with a timeout.

    Here's the threaded version of a timeout function:

    http://code.activestate.com/recipes/473878/

    However, do you need to read the stdout as it's coming in? Another solution may be to dump the output to a file and wait for the process to finish using p.wait().

    f = open('myprogram_output.txt','w')
    p = subprocess.Popen('myprogram.exe', stdout=f)
    p.wait()
    f.close()
    
    
    str = open('myprogram_output.txt','r').read()
    
    0 讨论(0)
  • 2020-11-21 05:16

    I add this problem to read some subprocess.Popen stdout. Here is my non blocking read solution:

    import fcntl
    
    def non_block_read(output):
        fd = output.fileno()
        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
        try:
            return output.read()
        except:
            return ""
    
    # Use example
    from subprocess import *
    sb = Popen("echo test && sleep 1000", shell=True, stdout=PIPE)
    sb.kill()
    
    # sb.stdout.read() # <-- This will block
    non_block_read(sb.stdout)
    'test\n'
    
    0 讨论(0)
提交回复
热议问题