Options for HTML scraping? [closed]

Answer 1

I've had some success with HtmlUnit, in Java. It's a simple framework for writing unit tests on web UI's, but equally useful for HTML scraping.

Answer 2

I would first find out if the site(s) in question provide an API server or RSS Feeds for access the data you require.

Answer 3

Yahoo! Query Language or YQL can be used alongwith jQuery, AJAX, JSONP to screen scrape web pages

Answer 4

For Perl, there's WWW::Mechanize.

Answer 5

I use Hpricot on Ruby. As an example this is a snippet of code that I use to retrieve all book titles from the six pages of my HireThings account (as they don't seem to provide a single page with this information):

pagerange = 1..6
proxy = Net::HTTP::Proxy(proxy, port, user, pwd)
proxy.start('www.hirethings.co.nz') do |http|
  pagerange.each do |page|
    resp, data = http.get "/perth_dotnet?page=#{page}" 
    if resp.class == Net::HTTPOK
      (Hpricot(data)/"h3 a").each { |a| puts a.innerText }
    end
  end
end 

It's pretty much complete. All that comes before this are library imports and the settings for my proxy.

Answer 6

Although it was designed for .NET web-testing, I've been using the WatiN framework for this purpose. Since it is DOM-based, it is pretty easy to capture HTML, text, or images. Recentely, I used it to dump a list of links from a MediaWiki All Pages namespace query into an Excel spreadsheet. The following VB.NET code fragement is pretty crude, but it works.

---

Sub GetLinks(ByVal PagesIE As IE, ByVal MyWorkSheet As Excel.Worksheet)

    Dim PagesLink As Link
    For Each PagesLink In PagesIE.TableBodies(2).Links
        With MyWorkSheet
            .Cells(XLRowCounterInt, 1) = PagesLink.Text
            .Cells(XLRowCounterInt, 2) = PagesLink.Url
        End With
        XLRowCounterInt = XLRowCounterInt + 1
    Next
End Sub