Options for HTML scraping? [closed]

Answer 1

You would be a fool not to use Perl.. Here come the flames..

Bone up on the following modules and ginsu any scrape around.

use LWP
use HTML::TableExtract
use HTML::TreeBuilder
use HTML::Form
use Data::Dumper

Answer 2

Another tool for .NET is MhtBuilder

Answer 3

Well, if you want it done from the client side using only a browser you have jcrawl.com. After having designed your scrapping service from the web application (http://www.jcrawl.com/app.html), you only need to add the generated script to an HTML page to start using/presenting your data.

All the scrapping logic happens on the the browser via JavaScript. I hope you find it useful. Click this link for a live example that extracts the latest news from Yahoo tennis.

Answer 4

I've had mixed results in .NET using SgmlReader which was originally started by Chris Lovett and appears to have been updated by MindTouch.

Answer 5

In the .NET world, I recommend the HTML Agility Pack. Not near as simple as some of the above options (like HTMLSQL), but it's very flexible. It lets you maniuplate poorly formed HTML as if it were well formed XML, so you can use XPATH or just itereate over nodes.

http://www.codeplex.com/htmlagilitypack

Answer 6

The Ruby world's equivalent to Beautiful Soup is why_the_lucky_stiff's Hpricot.