urllib

Random “[Errno -2] Name or service not known” errors

由 南笙酒味 提交于 2019-12-24 00:13:41
问题 I am populating a local database using a third party service. I have a list of urls (around 500). I am calling each url in a loop, and updating my database with the returned data. The code flow looks like this: for url in urllist: req = urllib.urlopen(url) data = json.loads(req.read()) req.close() #update the db using data here Whenever I run this piece of code, the script fails at random points with the error message "Name or service not known". This doesn't have anything with the urls

Downloading second file from ftp fails

由 眉间皱痕 提交于 2019-12-23 20:19:29
问题 I want to download multiple files from FTP in python. the my code works when I just download 1 file, but not works for more than one! import urllib urllib.urlretrieve('ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/00/00/PMC1790863.tar.gz', 'file1.tar.gz') urllib.urlretrieve('ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/00/00/PMC2329613.tar.gz', 'file2.tar.gz') An error say: Traceback (most recent call last): File "/home/ehsan/dev_center/bigADEVS-bknd/daemons/crawler/ftp_oa_crawler.py", line

Python: Urllib.urlopen nonnumeric port

由 ∥☆過路亽．° 提交于 2019-12-23 09:05:34
问题 for the following code theurl = "https://%s:%s@members.dyndns.org/nic/update?hostname=%s&myip=%s&wildcard=NOCHG&mx=NOCHG&backmx=NOCHG" % (username, password, hostname, theip) conn = urlopen(theurl) # send the request to the url print(conn.read()) # read the response conn.close() # close the connection i get the following error File "c:\Python31\lib\http\client.py", line 667, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) Any Ideas??? 回答1: You probably need to url

Python 3.5.1 urllib has no attribute request

由 落花浮王杯 提交于 2019-12-23 06:56:38
问题 I have tried import urllib.request or import urllib The path for my urllib is /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/__init__.py I am wondering where is urlopen, or is my python module pointing to the wrong file? 回答1: Use this: import urllib.request The reason is: With packages, like this, you sometimes need to explicitly import the piece you want. That way, the urllib module doesn't have to load everything up just because you wanted one small part. According

Python 3.5.1 urllib has no attribute request

由 。_饼干妹妹 提交于 2019-12-23 06:56:06
问题 I have tried import urllib.request or import urllib The path for my urllib is /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/__init__.py I am wondering where is urlopen, or is my python module pointing to the wrong file? 回答1: Use this: import urllib.request The reason is: With packages, like this, you sometimes need to explicitly import the piece you want. That way, the urllib module doesn't have to load everything up just because you wanted one small part. According

urllib for python 3

由 夙愿已清 提交于 2019-12-23 05:16:44
问题 This code in python3 is problematic： import urllib.request fhand=urllib.request.urlopen('http://www.py4inf.com/code/romeo.txt') print(fhand.read()) Its output is: b'But soft what light through yonder window breaks' b'It is the east and Juliet is the sun' b'Arise fair sun and kill the envious moon' b'Who is already sick and pale with grief' Why did I get b'...' ？ What could I do to get the right response？ The right text should be But soft what light through yonder window breaks It is the east

unbuffered urllib2.urlopen

由 烂漫一生 提交于 2019-12-23 02:32:32
问题 I have client for web interface to long running process. I'd like to have output from that process to be displayed as it comes. Works great with urllib.urlopen() , but it doesn't have timeout parameter. On the other hand with urllib2.urlopen() the output is buffered. Is there a easy way to disable that buffer? 回答1: A quick hack that has occurred to me is to use urllib.urlopen() with threading.Timer() to emulate timeout. But that's only quick and dirty hack. 回答2: urllib2 is buffered when you

TypeError: cannot concatenate 'str' and 'instance' objects (python urllib)

由 早过忘川 提交于 2019-12-23 00:39:33
问题 Writing a python program, and I came up with this error while using the urllib.urlopen function. Traceback (most recent call last): File "ChurchScraper.py", line 58, in <module> html = GetAllChurchPages() File "ChurchScraper.py", line 48, in GetAllChurchPages CPs = CPs + urllib.urlopen(url) TypeError: cannot concatenate 'str' and 'instance' objects url = 'http://website.com/index.php?cID=' + str(cID) CPs = CPs + urllib.urlopen(url) 回答1: urlopen(url) returns a file-like object. To obtain the

speeding up urlib.urlretrieve

由 大憨熊 提交于 2019-12-22 22:47:05
问题 I am downloading pictures from the internet, and as it turns out, I need to download lots of pictures. I am using a version of the following code fragment (actually looping through the links I intend to download and downloading the pictures : import urllib urllib.urlretrieve(link, filename) I am downloading roughly 1000 pictures every 15 minutes, which is awfully slow based on the number of pictures I need to download. For efficiency, I set a timeout every 5 seconds (still many downloads last

1.4requests的简单使用

由 [亡魂溺海] 提交于 2019-12-22 19:23:09
问题：为什么要学习requests，而不是urllib？ 1、requests的底层实现就是urllib 2、requests在python2 和python3中通用，方法完全一样 3、requests简单易用 4、requests能够自动帮助我们解压(gzip压缩的等)网页内容 Requests: 让 HTTP 服务人类 虽然Python的标准库中 urllib 模块已经包含了平常我们使用的大多数功能，但是它的 API 使用起来让人感觉不太好，而 Requests 自称 “HTTP for Humans”，说明使用更简洁方便。 Requests 继承了urllib的所有特性。Requests支持HTTP连接保持和连接池，支持使用cookie保持会话，支持文件上传，支持自动确定响应内容的编码，支持国际化的 URL 和 POST 数据自动编码。 requests 的底层实现其实就是 urllib Requests的文档非常完备，中文文档也相当不错。Requests能完全满足当前网络的需求，支持Python 2.6–3.5，而且能在PyPy下完美运行。 开源地址：https://github.com/kennethreitz/requests 中文文档 API： http://docs.python-requests.org/zh_CN/latest/index.html 安装方式
订阅 urllib