How do I properly write to FIFOs in Python?

Question

Something very strange is happening when I open FIFOs (named pipes) in Python for writing. Consider what happens when I try to open a FIFO for writing in a interactive interpreter:

>>> fifo_write = open('fifo', 'w')

The above line blocks until I open another interpreter and type the following:

>>> fifo_read = open('fifo', 'r')
>>> fifo.read()

I don't understand why I had to wait for the pipe to be opened for reading, but lets skip that. The above code will block until there's data available as expected. However let's say I go back to the first interpreter window and type:

>>> fifo_write.write("some testing data\n")
>>> fifo_write.flush()

The expected behavior is that on the second interpreter the call to read will return and we will see the data on the screen, except that is not happening to me. If I call os.fsync the following happens:

>>> import os
>>> fifo_write.flush()
>>> os.fsync(fifo_write.fileno())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

And the fifo reader is still waiting. However, if I call fifo_writer.close() then the data is flushed. If I use a shell command to feed the pipe:

$ echo "some data" > fifo

then the reader output is:

>>> fifo_read.read()
'some data\n'

Has anyone experienced this? If so is there a workaround for it? My current OS is Ubuntu 11.04 with Linux 2.6.38.

Answer 1

read() doesn't return until it reaches EOF.

You can try specifying the number of bytes you want read, like read(4). This will still block until enough bytes have been written, so the producer must write at least that many bytes and then call flush().

Answer 2

To avoid the need for flushing, open the file without buffering:

fifo_read = open('fifo', 'r', 0)

That will remove high-level buffering. Data go to the OS directly and, being a fifo, they never get actually written to disk but passed straight to the reader thru the fifo buffer, so you don't need to sync.

Of course, you should have created the fifo first with os.mkfifo() or mkfifo at the shell, as you pointed in a comment.