HtmlUnitDriver does not appear to be loading page

Question

I'm trying to get HtmlUnitDriver to work in my development environment. As a beginner, I've tried implementing the example from the following page using the latest selenium server jar: http://code.google.com/p/selenium/wiki/GettingStarted

Unfortunately, whenever I try to run this program, I get the following exception:

Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element with name: q
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '2.35.0', revision: 'c916b9d', time: '2013-08-12 15:42:01'
System info: os.name: 'Windows 7', os.arch: 'x86', os.version: '6.1', java.version: '1.6.0_16'
Driver info: driver.version: HtmlUnitDriver
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElementByName(HtmlUnitDriver.java:853)
    at org.openqa.selenium.By$ByName.findElement(By.java:292)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver$5.call(HtmlUnitDriver.java:1404)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver$5.call(HtmlUnitDriver.java:1)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.implicitlyWaitFor(HtmlUnitDriver.java:1094)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:1401)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:419)
    at Justin.Main.main(Main.java:30)

I've tried modifying my code to incorporate the fixes implemented here:

HtmlUnitDriver causes problems while getting an url

I've tried getting the URL of the page using driver.getCurrentUrl() after the call to driver.get("http://www.google.com"), but the string that is returned is about:blank.

Similar code in this example would definitely work if I were to run it using FirefoxDriver, but to meet requirements, I need my script to run headless with selenium (it is okay if it is run with a specific BrowserVersion so long as it is headless).

Any help would be greatly appreciated.

UPDATE:

This is the code I'm trying to run right now. I just want to see that I can get HtmlUnitDriver to work with something as simple as entering a search query into Google.

package Justin;


import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;


public class Main {
public static void main(final String[] args) {
    // Create a new instance of the html unit driver
    // Notice that the remainder of the code relies on the interface,
    // not the implementation.
    final WebDriver driver = new HtmlUnitDriver();
    ((HtmlUnitDriver)driver).setJavascriptEnabled(true);

    // And now use this to visit Google
    driver.get("http://www.google.com");

    try {
        Thread.sleep(30000);
    } catch (final InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    // Find the text input element by its name
    final WebElement element = driver.findElement(By.name("q"));

    // Enter something to search for
    element.sendKeys("Cheese!");

    // Now submit the form. WebDriver will find the form for us from the element
    element.submit();

    // Check the title of the page
    System.out.println("Page title is: " + driver.getTitle());

}
}

UPDATE 2:

This issue is very bizarre. I've tried the same code on my computer at home and it works perfectly fine with BrowserVersion set to Firefox or Internet Explorer. This issue is definitely being caused by the computer configuration at my workplace, although I still have no idea why this is happening.

Answer 1

In the example code given, it assumes the older Google page exists which has the search field with name=q.

The page no longer is labelled that way - try changing driver.findElement(By.name("q")) to driver.findElement(By.cssSelector("input[id=gbqfq]") and see if that fixes your issue - at the very least it should be able to input to the search bar once you get the page loaded up.

If you called driver.getCurrentUrl() immediately after trying to get the page, it may not have loaded completely and instead returned about:blank.

If that doesn't work, we can keep troubleshooting - it's harder to see what's actually happening with a headless browser, so you may want to switch to a FirefoxDriver temporarily to visualize exactly what's happening.

Answer 2

Ill try to give it a shot:

If it is a network problem, please download this tool to check if the port your are connecting to is in use: http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx

Very handy for these kind of situations.

For further debugging, try reading out the source. Put this line before the first element selector and after the page loaded.:

System.out.println(driver.getPageSource());

I wonder if the element is there..

Answer 3

Try using the Selenium IDE plugin with Firefox to see what it scrapes when it does it actions. You can export the code into java and then try to run that also. I would also step through the code on the both home and work computer to see if there are any differences. I have seen code work (when that element not found error happens) when being stepped through due to the DOM having time to fully load. Furthermore, try to instantiate and actual browser so you can visuallly see whats happening.

Also, verify version of all software your using between home and workstation.

Answer 4

1)Try to add

driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

before

final WebElement element = driver.findElement(By.name("q"));

2)or try

 try {
      WebElement element = driver.findElement(By.name("q"));
    } catch (Exception e) {
      Thread.sleep(1000);  
    }

Answer 5

If you require headless selenium, and firefox works with your test, try using the phantomjs webdriver.

Answer 6

Try using

HtmlUnitDriver driver = new HtmlUnitDriver();

Can you check if the page is loaded by checking

String pageTitle = driver.getTitle();