问题

I am trying to rotate my IP when doing web scraping but it seems is not working cause when I am checking the IP im doing this process is always the same. Hereunder the code I am using:

CODE:

import requests
from bs4 import BeautifulSoup
import random

headers = {'User-Agent': 'Mozilla/5.0 (Linux; Android 5.1.1; SM-G928X Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.83 Mobile Safari/537.36'}

def get_free_proxies():
    url = "https://free-proxy-list.net/"
    # get the HTTP response and construct soup object
    soup = BeautifulSoup(requests.get(url).content, "html.parser")
    proxies = list()
    for row in soup.find("table", attrs={"id": "proxylisttable"}).find_all("tr")[1:]:
        tds = row.find_all("td")
        try:
            ip = tds[0].text.strip()
            port = tds[1].text.strip()
            host = f"{ip}:{port}"
            proxies.append(host)
        except IndexError:
            continue
    return proxies

def get_session(proxies):
    #Construct an HTTP session
    session = requests.Session()
    #choose one random proxy
    proxy = random.choice(proxies)
    session.proxies = {"http": proxy, "https": proxy}
    #session.proxies.update(proxy)
    return session

proxies = get_free_proxies()

for i in range(5):
    session = get_session(proxies)
    
    print("Request page with IP:", session.get("http://icanhazip.com",timeout=1.5).text.strip())

And the output is always the same IP, is not been updated, and by the way is my computer IP

Does anyone knows what is failing?

Thank you all

回答1:

Perhaps you have set environment variable http_proxy and when sending a request, the proxy specified in this variable is used. To change this behavior you just need to set the attribute trust_env to False when creating a session

def get_session(proxies):
    #Construct an HTTP session
    session = requests.Session()
    #choose one random proxy
    proxy = random.choice(proxies)
    session.proxies = {"http": proxy, "https": proxy}
    session.trust_env = False
    return session

来源：https://stackoverflow.com/questions/61970930/proxy-configuration-is-not-working-in-python

标签

python