python os.mkfifo() for Windows

Question

Short version (if you can answer the short version it does the job for me, the rest is mainly for the benefit of other people with a similar task):

In python in Wind

Answer 1

On Windows, you are looking at (Named or Anonymous) Pipes.

A pipe is a section of shared memory that processes use for communication. The process that creates a pipe is the pipe server. A process that connects to a pipe is a pipe client. One process writes information to the pipe, then the other process reads the information from the pipe.

To work with Windows Pipes, you can use Python for Windows extensions (pywin32), or the Ctypes module. A special utility module, win32pipe, provides an interface to the win32 pipe API's. It includes implementations of the popen[234]() convenience functions.

See how-to-use-win32-apis-with-python and similar SO questions (not specific to Pipes, but points to useful info).

Answer 2

Following the two answers above, I accidentally bumped into the answer. os.pipe() does the job. Thank you for your answers.

I'm posting the complete code in case someone else is looking for this:

import subprocess
from threading import Thread
import time
import sys
import logging
import tempfile
import os

import game_playing_module

class Pusher(Thread):
    def __init__(self, source, dest, proc, name):
        Thread.__init__(self)
        self.source = source
        self.dest = dest
        self.name = name
        self.proc = proc

    def run(self):
        while (self.proc.poll()==None) and\
              (not self.source.closed) and (not self.dest.closed):
            line = self.source.readline()
            logging.info('%s: %s' % (self.name, line[:-1]))
            self.dest.write(line)
            self.dest.flush()

def get_reader_writer():
    fd_read, fd_write = os.pipe()
    return os.fdopen(fd_read, 'r'), os.fdopen(fd_write, 'w')

def connect(exe):
    logging.basicConfig(level=logging.DEBUG,\
                        format='%(message)s',\
                        filename=LOG_FILE_NAME,
                        filemode='w')

    program_to_grader_reader, program_to_grader_writer =\
                              get_reader_writer()

    grader_to_program_reader, grader_to_program_writer =\
                              get_reader_writer()

    p1 = subprocess.Popen(exe, shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)        

    old_stdin = sys.stdin
    old_stdout = sys.stdout

    sys.stdin = program_to_grader_reader
    sys.stdout = grader_to_program_writer

    push1 = Pusher(p1.stdout, program_to_grader_writer, p1, '1')
    push2 = Pusher(grader_to_program_reader, p1.stdin, p1, '2')

    push1.start()
    push2.start()

    game_playing_module.play()

    sys.stdin = old_stdin
    sys.stdout = old_stdout

    fil = file(LOG_FILE, 'r')
    data = fil.read()
    fil.close()
    return data

if __name__=='__main__':
    if len(sys.argv) != 2:
        print 'Usage: connect.py exe'
        print sys.argv
        exit()
    print sys.argv
    print connect(sys.argv[1])

Answer 3

For a cross-platform solution, I'd recommend building the file-like object on top of a socket on localhost (127.0.0.1) -- that's what IDLE does by default to solve a problem that's quite similar to yours.

Answer 4

os.pipe() returns an anonymous pipe, or a named pipe on Windows, which is very lightweight and efficient.

TCP sockets (as suggested by user1495323) are more heavyweight: you can see them with netstat for example, and each one requires a port number, and the number of available ports is limited to 64k per peer (e.g. 64k from localhost to localhost).

On the other hand, named pipes (on Windows) are limited because:

• You can't use select() for nonblocking I/O on Windows, because they're not sockets.
• There's no apparent way to read() with a timeout, and
• Even making them non-blocking is difficult.

And sockets can be wrapped in Python-compatible filehandles using makefile(), which allows them to be used to redirect stdout or stderr. This makes this an attractive option for some use cases, such as sending stdout from one thread to another.

A socket can be constructed with an automatically-assigned port number like this (based on the excellent Python socket HOWTO):

with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as input_socket:
    # Avoid socket exhaustion by setting SO_REUSEADDR <https://stackoverflow.com/a/12362623/648162>:
    input_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # localhost doesn't work if the definition is missing from the hosts file,
    # and 127.0.0.1 only works with IPv4 loopback, but socket.gethostname()
    # should always work:
    input_socket.bind((socket.gethostname(), 0))
    random_port_number = input_socket.getsockname()[1]
    input_socket.listen(1)

    # Do something with input_socket, for example pass it to another thread.

    output_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    # close() should not strictly be necessary here, but since connect() could fail, it avoids leaking fds
    # in that case. "If a file descriptor is given, it is closed when the returned I/O object is closed".
    with output_socket:
        output_socket.connect((socket.gethostname(), random_port_number))

The user of input_socket (e.g. another thread) can then do:

with input_socket:
    while True:
        readables, _, _ = select.select([input_socket], [], [input_socket], 1.0)

        if len(readables) > 0:
            input_conn, addr = self.input_socket.accept()
            break

    with input_conn:
        while True:
            readables, _, errored = select.select([input_conn], [], [input_conn], 1.0)
            if len(errored) > 0:
                print("connection errored, stopping")
                break

            if len(readables) > 0:
                read_data = input_conn.recv(1024)
                if len(read_data) == 0:
                    print("connection closed, stopping")
                    break
                else:
                    print(f"read data: {read_data!r}")