multiprocessing.pool.MaybeEncodingError: 'TypeError(“cannot serialize '_io.BufferedReader' object”,)'

Question

Why does the code below work only with multiprocessing.dummy, but not with simple multiprocessing.

import urllib.request
#from mult         


        

Answer 1

The http.client.HTTPResponse-object you get back from urlopen() has a _io.BufferedReader-object attached, and this object cannot be pickled.

pickle.dumps(urllib.request.urlopen('http://www.python.org').fp)
Traceback (most recent call last):
...
    pickle.dumps(urllib.request.urlopen('http://www.python.org').fp)
TypeError: cannot serialize '_io.BufferedReader' object

multiprocessing.Pool will need to pickle (serialize) the results to send it back to the parent process and this fails here. Since dummy uses threads instead of processes, there will be no pickling, because threads in the same process share their memory naturally.

A general solution to this TypeError is:

1. read out the buffer and save the content (if needed)
2. remove the reference to '_io.BufferedReader' from the object you are trying to pickle

In your case, calling .read() on the http.client.HTTPResponse will empty and remove the buffer, so a function for converting the response into something pickleable could simply do this:

def read_buffer(response):
    response.text = response.read()
    return response

Example:

r = urllib.request.urlopen('http://www.python.org')
r = read_buffer(r)
pickle.dumps(r)
# Out: b'\x80\x03chttp.client\nHTTPResponse\...

Before you consider this approach, make sure you really want to use multiprocessing instead of multithreading. For I/O-bound tasks like you have it here, multithreading would be sufficient, since most of the time is spend in waiting (no need for cpu-time) for the response anyway. Multiprocessing and the IPC involved also introduces substantial overhead.