urllib

HTTPS POST request Python, returning .csv

由 风格不统一 提交于 2019-12-06 07:57:08
I want to make a post request to a HTTPS-site that should respond with a .csv file. I have this Python code: try: #conn = httplib.HTTPSConnection(host="www.site.com", port=443) => Gives an BadStatusLine: ' ' error conn = httplib.HTTPConnection("www.site.com"); params = urllib.urlencode({'val1':'123','val2':'abc','val3':'1b3'}) conn.request("POST", "/nps/servlet/exportdatadownload", params) content = conn.getresponse() print content.reason, content.status print content.read() conn.close() except: import sys print sys.exc_info()[:2] Output: Found 302 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0/

urllib: Get name of file from direct download link

由 懵懂的女人 提交于 2019-12-06 07:56:50
问题 Python 3. Probably need to use urllib to do this, I need to know how to send a request to a direct download link, and get the name of the file it attempts to save. (As an example, a KSP mod from CurseForge: https://kerbal.curseforge.com/projects/mechjeb/files/2355387/download) Of course, the file ID ( 2355387 ) will be changed. It could be from any project, but always on CurseForge. (If that makes a difference on the way it's downloaded.) That example link results in the file: How can I

Setting proxy to urllib.request (Python3)

由 五迷三道 提交于 2019-12-06 05:57:56
问题 How can I set proxy for the last urllib in Python 3. I am doing the next from urllib import request as urlrequest ask = urlrequest.Request(url) # note that here Request has R not r as prev versions open = urlrequest.urlopen(req) open.read() I tried adding proxy as follows : ask=urlrequest.Request.set_proxy(ask,proxies,'http') However I don't know how correct it is since I am getting the next error: 336 def set_proxy(self, host, type): --> 337 if self.type == 'https' and not self._tunnel_host:

How to read the content of an website?

由 倖福魔咒の 提交于 2019-12-06 05:19:01
I'm new on web-crawler using python 2.7. 1. Background Now, I want to collect useful data from AQICN.org which is a great website offering the air quality data all over the world. I want to use python to get all China's sites data per hour. But I'm stuck right now. 2. My trouble Take this website( http://aqicn.org/city/shenyang/usconsulate/ ) for example. This page offer the air pollution and meteorology parameters of a U.S Consulate in China. Using code like this, I can't get useful information. import urllib from bs4 import BeautifulSoup import re import json html_aqi = urllib.urlopen("http:

Python Urllib UrlOpen Read

由 梦想与她 提交于 2019-12-06 04:36:08
Say I am retrieving a list of Urls from a server using Urllib2 library from Python. I noticed that it took about 5 seconds to get one page and it would take a long time to finish all the pages I want to collect. I am thinking out of those 5 seconds. Most of the time was consumed on the server side and I am wondering could I just start using the threading library. Say 5 threads in this case, then the average time could be dramatically increased. Maybe 1 or 2 seconds in each page. (might make the server a bit busy). How could I optimize the number of threads so I could get a legit speed and not

Python 3 urllib with self-signed certificates

由 三世轮回 提交于 2019-12-06 04:16:33
I'm attempting to download some data from an internal server using Python. Since it's internal, it uses a self-signed certificate. (We don't want to pay Verisign for servers that will never appear "in the wild.") The Python 2.6 version of the code worked fine. response = urllib2.urlopen(URL) data = csv.reader(response) I'm now trying to update to Python 3.4 (long story, don't ask.) However, using Python 3's urllib fails: response = urllib.request.urlopen(URL) It throws a CERTIFICATE_VERIFY_FAILED error. urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify

Write contents of URL request to file

由 旧城冷巷雨未停 提交于 2019-12-06 02:18:43
I am trying to fetch a list from a php file using python and save it to a file: import urllib.request page = urllib.request.urlopen('http://crypto-bot.hopto.org/server/list.php') f = open("test.txt", "w") f.write(str(page)) f.close() print(page.read()) Output on screen (divided onto four lines for readability): ALF\nAMC\nANC\nARG\nBQC\nBTB\nBTE\nBTG\nBUK\nCAP\nCGB\nCLR\nCMC\nCRC\nCSC\nDGC\n DMD\nELC\nEMD\nFRC\nFRK\nFST\nFTC\nGDC\nGLC\nGLD\nGLX\nHBN\nIXC\nKGC\nLBW\nLKY\n LTC\nMEC\nMNC\nNBL\nNEC\nNMC\nNRB\nNVC\nPHS\nPPC\nPXC\nPYC\nQRK\nSBC\nSPT\nSRC\n STR\nTRC\nWDC\nXPM\nYAC\nYBC\nZET\n Output

urllib3 maxretryError

由 可紊 提交于 2019-12-06 00:24:52
问题 I have just started using urllib3, and I am running into a problem straightaway. According to their manuals, I started off with the simple example: Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib3 >>> >>> http = urllib3.PoolManager() >>> r = http.request('GET', 'http://google.com/') I get thrown the following error: Traceback (most recent call last): File "<stdin>", line 1, in

Python3: urllib.error.HTTPError: HTTP Error 403: Forbidden

由 泄露秘密 提交于 2019-12-05 23:26:43
Please, Help me! I am using Python3.3 and this code: import urllib.request import sys Open_Page = urllib.request.urlopen( "http://wowcircle.com" ).read().decode().encode('utf-8') And I take this: Traceback (most recent call last): File "C:\Users\1\Desktop\WCLauncer\reg.py", line 5, in <module> "http://forum.wowcircle.com" File "C:\Python33\lib\urllib\request.py", line 156, in urlopen return opener.open(url, data, timeout) File "C:\Python33\lib\urllib\request.py", line 475, in open response = meth(req, response) File "C:\Python33\lib\urllib\request.py", line 587, in http_response 'http',

python urllib error - AttributeError: 'module' object has no attribute 'request'

由 本秂侑毒 提交于 2019-12-05 22:34:28
I am trying out a tutorial code which fetches the html code form a website and prints it. I'm using python 3.4.0 on ubuntu. The code: import urllib.request page = urllib.request.urlopen("http://www.brainjar.com/java/host/test.html") text = page.read().decode("utf8") print(text) I saw previous solutions and tried them, I also tried importing only urllib but it still doesn't work. The error message displayed is as shown: Traceback (most recent call last): File "string.py", line 1, in <module> import urllib.request File "/usr/lib/python3.4/urllib/request.py", line 88, in <module> import http
订阅 urllib