Options for HTML scraping? [closed]

Answer 1

I found HTMLSQL to be a ridiculously simple way to screenscrape. It takes literally minutes to get results with it.

The queries are super-intuitive - like:

SELECT title from img WHERE $class == 'userpic'

There are now some other alternatives that take the same approach.

Answer 2

Scraping Stack Overflow is especially easy with Shoes and Hpricot.

require 'hpricot'

Shoes.app :title => "Ask Stack Overflow", :width => 370 do
  SO_URL = "http://stackoverflow.com"
  stack do
    stack do
      caption "What is your question?"
      flow do
        @lookup = edit_line "stackoverflow", :width => "-115px"
        button "Ask", :width => "90px" do
          download SO_URL + "/search?s=" + @lookup.text do |s|
            doc = Hpricot(s.response.body)
            @rez.clear()
            (doc/:a).each do |l|
              href = l["href"]
              if href.to_s =~ /\/questions\/[0-9]+/ then
                @rez.append do
                  para(link(l.inner_text) { visit(SO_URL + href) })
                end
              end
            end
            @rez.show()
          end
        end
      end
    end
    stack :margin => 25 do
      background white, :radius => 20
      @rez = stack do
      end
    end
    @rez.hide()
  end
end

Answer 3

Implementations of the HTML5 parsing algorithm: html5lib (Python, Ruby), Validator.nu HTML Parser (Java, JavaScript; C++ in development), Hubbub (C), Twintsam (C#; upcoming).

Answer 4

BeautifulSoup is a great way to go for HTML scraping. My previous job had me doing a lot of scraping and I wish I knew about BeautifulSoup when I started. It's like the DOM with a lot more useful options and is a lot more pythonic. If you want to try Ruby they ported BeautifulSoup calling it RubyfulSoup but it hasn't been updated in a while.

Other useful tools are HTMLParser or sgmllib.SGMLParser which are part of the standard Python library. These work by calling methods every time you enter/exit a tag and encounter html text. They're like Expat if you're familiar with that. These libraries are especially useful if you are going to parse very large files and creating a DOM tree would be long and expensive.

Regular expressions aren't very necessary. BeautifulSoup handles regular expressions so if you need their power you can utilize it there. I say go with BeautifulSoup unless you need speed and a smaller memory footprint. If you find a better HTML parser on Python, let me know.

Answer 5

The Python lxml library acts as a Pythonic binding for the libxml2 and libxslt libraries. I like particularly its XPath support and pretty-printing of the in-memory XML structure. It also supports parsing broken HTML. And I don't think you can find other Python libraries/bindings that parse XML faster than lxml.

Answer 6

The templatemaker utility from Adrian Holovaty (of Django fame) uses a very interesting approach: You feed it variations of the same page and it "learns" where the "holes" for variable data are. It's not HTML specific, so it would be good for scraping any other plaintext content as well. I've used it also for PDFs and HTML converted to plaintext (with pdftotext and lynx, respectively).