Proxies with Python 'Requests' module

后端 未结 10 810
傲寒
傲寒 2020-11-22 12:13

Just a short, simple one about the excellent Requests module for Python.

I can\'t seem to find in the documentation what the variable \'proxies\' should contain. Whe

相关标签:
10条回答
  • 2020-11-22 12:30

    You can refer to the proxy documentation here.

    If you need to use a proxy, you can configure individual requests with the proxies argument to any request method:

    import requests
    
    proxies = {
      "http": "http://10.10.1.10:3128",
      "https": "https://10.10.1.10:1080",
    }
    
    requests.get("http://example.org", proxies=proxies)
    

    To use HTTP Basic Auth with your proxy, use the http://user:password@host.com/ syntax:

    proxies = {
        "http": "http://user:pass@10.10.1.10:3128/"
    }
    
    0 讨论(0)
  • 2020-11-22 12:34

    The proxies' dict syntax is {"protocol":"ip:port", ...}. With it you can specify different (or the same) proxie(s) for requests using http, https, and ftp protocols:

    http_proxy  = "http://10.10.1.10:3128"
    https_proxy = "https://10.10.1.11:1080"
    ftp_proxy   = "ftp://10.10.1.10:3128"
    
    proxyDict = { 
                  "http"  : http_proxy, 
                  "https" : https_proxy, 
                  "ftp"   : ftp_proxy
                }
    
    r = requests.get(url, headers=headers, proxies=proxyDict)
    

    Deduced from the requests documentation:

    Parameters:
    method – method for the new Request object.
    url – URL for the new Request object.
    ...
    proxies – (optional) Dictionary mapping protocol to the URL of the proxy.
    ...


    On linux you can also do this via the HTTP_PROXY, HTTPS_PROXY, and FTP_PROXY environment variables:

    export HTTP_PROXY=10.10.1.10:3128
    export HTTPS_PROXY=10.10.1.11:1080
    export FTP_PROXY=10.10.1.10:3128
    

    On Windows:

    set http_proxy=10.10.1.10:3128
    set https_proxy=10.10.1.11:1080
    set ftp_proxy=10.10.1.10:3128
    

    Thanks, Jay for pointing this out:
    The syntax changed with requests 2.0.0.
    You'll need to add a schema to the url: https://2.python-requests.org/en/latest/user/advanced/#proxies

    0 讨论(0)
  • 2020-11-22 12:36

    8 years late. But I like:

    import os
    import requests
    
    os.environ['HTTP_PROXY'] = os.environ['http_proxy'] = 'http://http-connect-proxy:3128/'
    os.environ['HTTPS_PROXY'] = os.environ['https_proxy'] = 'http://http-connect-proxy:3128/'
    os.environ['NO_PROXY'] = os.environ['no_proxy'] = '127.0.0.1,localhost,.local'
    
    r = requests.get('https://example.com')  # , verify=False
    
    0 讨论(0)
  • 2020-11-22 12:39

    here is my basic class in python for the requests module with some proxy configs and stopwatch !

    import requests
    import time
    class BaseCheck():
        def __init__(self, url):
            self.http_proxy  = "http://user:pw@proxy:8080"
            self.https_proxy = "http://user:pw@proxy:8080"
            self.ftp_proxy   = "http://user:pw@proxy:8080"
            self.proxyDict = {
                          "http"  : self.http_proxy,
                          "https" : self.https_proxy,
                          "ftp"   : self.ftp_proxy
                        }
            self.url = url
            def makearr(tsteps):
                global stemps
                global steps
                stemps = {}
                for step in tsteps:
                    stemps[step] = { 'start': 0, 'end': 0 }
                steps = tsteps
            makearr(['init','check'])
            def starttime(typ = ""):
                for stemp in stemps:
                    if typ == "":
                        stemps[stemp]['start'] = time.time()
                    else:
                        stemps[stemp][typ] = time.time()
            starttime()
        def __str__(self):
            return str(self.url)
        def getrequests(self):
            g=requests.get(self.url,proxies=self.proxyDict)
            print g.status_code
            print g.content
            print self.url
            stemps['init']['end'] = time.time()
            #print stemps['init']['end'] - stemps['init']['start']
            x= stemps['init']['end'] - stemps['init']['start']
            print x
    
    
    test=BaseCheck(url='http://google.com')
    test.getrequests()
    
    0 讨论(0)
提交回复
热议问题