Python multiprocessing: synchronizing file-like object

Question

I\'m trying to make a file like object which is meant to be assigned to sys.stdout/sys.stderr during testing to provide deterministic output. It\'s not meant to be fast, just re

Answer 1

The solution came in two parts. I've successfully run the test program 200 thousand times without any change in output.

The easy part was to use multiprocessing.current_process()._identity to sort the messages. This is not a part of the published API, but it is a unique, deterministic identifier of each process. This fixed the problem with PIDs wrapping around and giving a bad ordering of output.

The other part of the solution was to use multiprocessing.Manager().Queue() rather than the multiprocessing.Queue. This fixes problem #2 above because the manager lives in a separate Process, and so avoids some of the bad special cases when using a Queue from the owning process. #3 is fixed because the Queue is fully exhausted and the feeder thread dies naturally before python starts shutting down and closes stdin.