How can I get dynamically web content using Perl?

Question

This is kind of tricky. There is this webpage which, I am guessing, uses some kind of AJAX to pull out content based on the search query. When I fetch the page using get in

Answer 1

Take a look at Selenium RC and the WWW::Selenium module in Perl. With them you can control a real web browser.

Another option is WWW::HtmlUnit which uses the HtmlUnit Java library to execute the JavaScript without a web browser. WWW::HtmlUnit uses Inline::Java to give Perl access to the library. I have found that when installing, it is best to say No to the question "Do you wish to build the JNI extension?".

Answer 2

If you are writing tests that need to check the rendered page, you can have a look at Schwern's javascript-tap-harness, which works with Selenium and handles all the scaffolding.

I also found Using WWW::Selenium To Test Or Automate An Ajax Website pretty useful.