unbuffered urllib2.urlopen

Question

I have client for web interface to long running process. I'd like to have output from that process to be displayed as it comes. Works great with urllib.urlopen(), but it doesn't have timeout parameter. On the other hand with urllib2.urlopen() the output is buffered. Is there a easy way to disable that buffer?

Answer 1

A quick hack that has occurred to me is to use urllib.urlopen() with threading.Timer() to emulate timeout. But that's only quick and dirty hack.

Answer 2

urllib2 is buffered when you just call read()

you could define a size to read and therefore disable buffering.

for example:

import urllib2

CHUNKSIZE = 80

r = urllib2.urlopen('http://www.python.org')
while True:
    chunk = r.read(CHUNKSIZE)
    if not chunk:
        break
    print(chunk)

this would print the response after each chunk is read from the socket, not buffer until the entire response is received.