HTMLUnit doesn't wait for Javascript

Question

I have a GWT based page that I would like to create an HTML snapshot for it using HtmlUnit. The page loads using Ajax/JavaScript information on a product, so for about 1 sec

Answer 1

None of the so far provided solutions worked for me. I ended up with Dan Alvizu's solution + my own hack:

private WebClient webClient = new WebClient();

public void scrapPage() {
    makeWebClientWaitThroughJavaScriptLoadings();
    HtmlPage page = login();
    //do something that causes JavaScript loading
    waitOutLoading(page);
}

private void makeWebClientWaitThroughJavaScriptLoadings() {
    webClient.setAjaxController(new AjaxController(){
        @Override
        public boolean processSynchron(HtmlPage page, WebRequest request, boolean async)
        {
            return true;
        }
    });
}

private void waitOutLoading(HtmlPage page) {
    while(page.asText().contains("Please wait while loading!")){
        webClient.waitForBackgroundJavaScript(100);
    }
}

Needless to say, "Please wait while loading!" should be replaced with whatever text is shown while your page is loading. If there is no text, maybe there is a way to check for existence of some gif (if that is used). Of course, you could simply provide a big enough milliseconds value if you're feeling adventurous.

Answer 2

I believe by default NicelyResynchronizingAjaxController will only resynchronize AJAX calls that were caused by a user action, by tracking which thread it originated from. Perhaps the GWT generated JavaScript is being called by some other thread which NicelyResynchronizingAjaxController does not want to wait for.

Try declaring your own AjaxController to synchronize with everything regardless of originating thread:

webClient.setAjaxController(new AjaxController(){
    @Override
    public boolean processSynchron(HtmlPage page, WebRequest request, boolean async)
    {
        return true;
    }
});

Answer 3

Thank you for responding. I actually should have reported this sooner that I have found the solution myself. Apparently when initialising WebClient with FF:

WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6);

It seem to be working. When initialising WebClient with the default constructor it uses IE7 by default and I guess FF has better support for Ajax and is the recommended emulator to use.

Answer 4

As documentation states, waitForBackgroundJavaScript is experimental:

Experimental API: May be changed in next release and may not yet work perfectly!

The next approach has always worked for me, regardless of the BrowserVersion used:

int tries = 5;  // Amount of tries to avoid infinite loop
while (tries > 0 && aCondition) {
    tries--;
    synchronized(page) {
        page.wait(2000);  // How often to check
    }
}

Note aCondition is whatever you're checking for. EG:

page.getElementById("loading-text-element").asText().equals("Loading...")