HTMLUnit : super slow execution?

Question

I have been using HTMLUnit . It suits my requirements well. But it seems to be extremely slow. for example : I have automated the following scenario using HTMLUnit

Answer 1

For the current htmlUnit 2.13, setting options is slightly different from what maxmax has provided:

final WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setCssEnabled(false);//if you don't need css
webClient.getOptions().setJavaScriptEnabled(false);//if you don't need js
HtmlPage page = webClient.getPage("http://XXX.xxx.xx");
...

In my own test, this is 8 times faster than the default options.(Note that this could be webpage-dependent)

Answer 2

• Be sure to use latest htmlunit version (2.9). I had a performance boost from previous version.

I get your example done within 20s, or 40s depending options i set. As i can't see the webClient initialisation, i guess maybe it could be the problem.

Here's my initialisation for a 20s treatment :

WebClient client = new WebClient(BrowserVersion.FIREFOX_3_6);
    client.setTimeout(60000);
    client.setRedirectEnabled(true);
    client.setJavaScriptEnabled(true);
    client.setThrowExceptionOnFailingStatusCode(false);
    client.setThrowExceptionOnScriptError(false);
    client.setCssEnabled(false);
    client.setUseInsecureSSL(true);

Answer 3

I recommend also to set a time limit to the javascript:

   client.setJavaScriptTimeout(30000); //e.g. 30s