Whats the best screen scraping language? [closed]

Answer 1

Ruby is pretty great !... try its hpricot/mechanize

Answer 2

Groovy is very good.

Example: http://froth-and-java.blogspot.com/2007/06/html-screen-scraping-with-groovy.html

Groovy and HtmlUnit is also a very good match: http://groovy.codehaus.org/Testing+Web+Applications Htmlunit will simulate a full browser with Javascript support.

Answer 3

C# is more than suitable for your screen scraping needs. .NET's Regex functionality is really nice. However, with such a simple task, you'll be hard to find a language that doesn't do what you want relatively easily. Considering you're already programming in C#, I'd say stick with that.

The built in screen scraping functionality is also top notch.

Answer 4

We use Groovy with NekoHTML. (Also note that you can now run Groovy on Google App Engine.)

Here is some example, runnable code on the Keplar blog:

Better competitive intelligence through scraping with Groovy

Answer 5

I second the recommendation for python (or Beautiful Soup). I'm currently in the middle of a small screen-scraping project using python, and python 3's automatic handling of things like cookie authentication (through CookieJar and urllib) are greatly simplifying things. Python supports all of the more advanced features you might need (like regexes), as well as having the benefit of being able to handle projects like this quickly (not too much overhead in dealing with low level stuff). It's also relatively cross-platform.

Answer 6

IMO Perl's built in regular expression functionality and ability to manipulate text would make it a pretty good contender for screen scraping.