How to know if my Custom HTTP headers are being passed?

问题

So, I have been struggling with passing custom HTTP header for some time now.

I am creating a script (Python) to Open a URL with Custom headers like

    {'Referer': 'https://google.com', 'X-Forwarded-For': '47.29.76.109', 

'User-Agent': 'Mozilla/5.0 (Linux; Android 7.1.1; CPH1723 Build/N6F26Q; 

wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 

Mobile Safari/537.36', 'existing_proxy_port_to_use': '8090'}

I have been using BrowserMob-Proxy for this but I am unable to see the effect when I try checking Network field in Inspect of Google Chrome.

CODE:

def automation():
    headers = pd.read_excel('Database/header.xlsx')
    for i in range(0,headers.shape[0]):
        dict = {}
        header = headers.loc[i]
        dict['Referer'] = header['Referrer']
        dict[header['Option']] = header['IP']
        dict['User-Agent'] = header['USERAGENT']
        dict['existing_proxy_port_to_use'] = "8090"
        print(dict)

        URL = 'xyz'
        data = pd.read_csv('Database/data.csv')
        server = Server(path="./browsermob-proxy/bin/browsermob-proxy", options=dict)
        server.start()
        proxy = server.create_proxy()
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy)) #Configure chrome options
        driver = webdriver.Chrome(chrome_options=chrome_options,executable_path='/home/.../chromedriver')
        proxy.new_har("google")
        for j in range(0,data.shape[0]):
            datum = data.loc[j]
            print(datum)
            driver.get(URL)
        driver.quit()
        server.stop()   
    return None    

automation()

I am reading these Parameters from header file and using Selenium to fill the Google Form.

So, Please help me in knowing how to pass the headers correctly and how to know if they are working.

回答1:

I solved the problem of passing the header by removing the Browsermob-proxy and instead using seleniumwire and use its driver._client.set_header_overrides(headers=dict_headers) to override the default HTTP headers.

def automation():

    headers = pd.read_excel('Database/header.xlsx')
    data = pd.read_csv('Database/data.csv')

    for i in range(0,headers.shape[0]):
        dict_headers = {}
        header = headers.loc[i]
        dict_headers['Referer'] = header['Referrer']
        dict_headers[header['Option']] = header['IP']
        dict_headers['User-Agent'] = header['USERAGENT']

        URL = 'xyz'

        user_agent = "user-agent="+header['USERAGENT']
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument(user_agent) 
        driver = webdriver.Chrome(
                chrome_options=chrome_options,
                executable_path='/home/.../chromedriver')

        driver._client.set_header_overrides(headers=dict_headers)
        datum = data.loc[i]
        driver.get(URL)
        driver.quit()
    return None    
automation()

来源：https://stackoverflow.com/questions/54456944/how-to-know-if-my-custom-http-headers-are-being-passed

标签

python

selenium

http-headers

browsermob-proxy