Options for HTML scraping? [closed]

Answer 1

I've used Beautiful Soup a lot with Python. It is much better than regular expression checking, because it works like using the DOM, even if the HTML is poorly formatted. You can quickly find HTML tags and text with simpler syntax than regular expressions. Once you find an element, you can iterate over it and its children, which is more useful for understanding the contents in code than it is with regular expressions. I wish Beautiful Soup existed years ago when I had to do a lot of screenscraping -- it would have saved me a lot of time and headache since HTML structure was so poor before people started validating it.

Answer 2

I've also had great success using Aptana's Jaxer + jQuery to parse pages. It's not as fast or 'script-like' in nature, but jQuery selectors + real JavaScript/DOM is a lifesaver on more complicated (or malformed) pages.

Answer 3

Python has several options for HTML scraping in addition to Beatiful Soup. Here are some others:

• mechanize: similar to perl WWW:Mechanize. Gives you a browser like object to ineract with web pages
• lxml: Python binding to libwww. Supports various options to traverse and select elements (e.g. XPath and CSS selection)
• scrapemark: high level library using templates to extract informations from HTML.
• pyquery: allows you to make jQuery like queries on XML documents.
• scrapy: an high level scraping and web crawling framework. It can be used to write spiders, for data mining and for monitoring and automated testing

Answer 4

'Simple HTML DOM Parser' is a good option for PHP, if your familiar with jQuery or JavaScript selectors then you will find yourself at home.

Find it here

There is also a blog post about it here.

Answer 5

I have used LWP and HTML::TreeBuilder with Perl and have found them very useful.

LWP (short for libwww-perl) lets you connect to websites and scrape the HTML, you can get the module here and the O'Reilly book seems to be online here.

TreeBuilder allows you to construct a tree from the HTML, and documentation and source are available in HTML::TreeBuilder - Parser that builds a HTML syntax tree.

There might be too much heavy-lifting still to do with something like this approach though. I have not looked at the Mechanize module suggested by another answer, so I may well do that.

Answer 6

There is this solution too: netty HttpClient