Pass the user-agent through webdriver in Selenium

Question

I am working on a website scraping project using Selenium in Python. When I open the homepage through a browser, it opens properly.

But, when I try to open the webpa

Answer 1

Changing the user agent in the python version of webdriver is done by altering your browser's profile. I have only done this for webdriver.Firefox() by passing a profile parameter. You need to do the following:

from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","your_user_agent_string")
driver=webdriver.Firefox(profile)

Every time you wish to change the user agent you will need to restart your web browser (i.e. call driver=webdriver.Firefox(profile) again)

If you are unsure to what your user agent string is do a search for "what is my user agent" on a browser that displays the page properly and just copy and paste that one.

Hope that sorts it.

Answer 2

Assuming the user-agent is the problem, in Java you can modify it like this:

FirefoxProfile profile = new FirefoxProfile();
profile.addAdditionalPreference("general.useragent.override", "some UA string");
WebDriver driver = new FirefoxDriver(profile);

See documentation here.