hangs on open url with urllib (python3)

问题

I try to open url with python3:

import urllib.request
fp = urllib.request.urlopen("http://lebed.com/")

mybytes = fp.read()    
mystr = mybytes.decode("utf8")
fp.close()

print(mystr)

But it hangs on second line. What's the reason of this problem and how to fix it?

回答1:

I suppose the reason is that the url does not support robot visiting a site visit. You need to fake a browser visit by sending browser headers along with your request

import urllib.request
url = "http://lebed.com/"
req = urllib.request.Request(
    url, 
    data=None, 
    headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
)
f = urllib.request.urlopen(req)

Tried this one on my system and it works.

回答2:

Agree with Arpit Solanki. Shown output for a failed request vs successful.

Failed
    GET / HTTP/1.1
    Accept-Encoding: identity
    Host: www.lebed.com
    Connection: close
    User-Agent: Python-urllib/3.5

Success
    GET / HTTP/1.1
    Accept-Encoding: identity
    Host: www.lebed.com
    Connection: close
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36

来源：https://stackoverflow.com/questions/45768441/hangs-on-open-url-with-urllib-python3