How can I screen scrape with Perl?

Question

I need to display some values that are stored in a website, for that I need to scrape the website and fetch the content from the table. Any ideas?

Answer 1

If you're familiar with XPath, you can also use HTML::TreeBuilder::XPath. And if you're not... well you should be ;--)

Answer 2

For similar Stackoverflow questions have a look at....

• How can I extract URLs from a web page in Perl
• How can I extract XML of a website and save in a file using Perl’s LWP?

I do like using pQuery for things like this however Web::Scraper does look interesting.

Answer 3

I have used HTML Table Extract in the past. I personally find it a bit clumsy to use, but maybe I did not understand the object model well. I usually use this part of the manual to examine the data:

 use HTML::TableExtract;
 $te = HTML::TableExtract->new();
 $te->parse($html_string);

     # Examine all matching tables
     foreach $ts ($te->tables) {
       print "Table (", join(',', $ts->coords), "):\n";
       foreach $row ($ts->rows) {
          print join(',', @$row), "\n";
       }
     }`

Answer 4

Take a look at the magical Web::Scraper, it's THE tool for web scraping.

Answer 5

I don't mean to drag up a dead thread but anyone googling across this thread should also checkout WWW::Scripter - 'For scripting web sites that have scripts'

happy remote data aggregating ;)

Answer 6

Although I've generally done this with LWP/LWP::Simple, the current 'preferred' module for any sort of webpage scraping in Perl is WWW::Mechanize.