Options for HTML scraping? [closed]

Answer 1

I like Google Spreadsheets' ImportXML(URL, XPath) function.

It will repeat cells down the column if your XPath expression returns more than one value.

You can have up to 50 importxml() functions on one spreadsheet.

RapidMiner's Web Plugin is also pretty easy to use. It can do posts, accepts cookies, and can set the user-agent.

Answer 2

I know and love Screen-Scraper.

Screen-Scraper is a tool for extracting data from websites. Screen-Scraper automates:

* Clicking links on websites
* Entering data into forms and submitting
* Iterating through search result pages
* Downloading files (PDF, MS Word, images, etc.)

Common uses:

* Download all products, records from a website
* Build a shopping comparison site
* Perform market research
* Integrate or migrate data

Technical:

* Graphical interface--easy automation
* Cross platform (Linux, Mac, Windows, etc.)
* Integrates with most programming languages (Java, PHP, .NET, ASP, Ruby, etc.)
* Runs on workstations or servers

Three editions of screen-scraper:

* Enterprise: The most feature-rich edition of screen-scraper. All capabilities are enabled.
* Professional: Designed to be capable of handling most common scraping projects.
* Basic: Works great for simple projects, but not nearly as many features as its two older brothers.

Answer 3

Another option for Perl would be Web::Scraper which is based on Ruby's Scrapi. In a nutshell, with nice and concise syntax, you can get a robust scraper directly into data structures.

Answer 4

In Java, you can use TagSoup.

Answer 5

Why has no one mentioned JSOUP yet for Java? http://jsoup.org/

Answer 6

You probably have as much already, but I think this is what you are trying to do:

from __future__ import with_statement
import re, os

profile = ""

os.system('wget --no-cookies --header "Cookie: soba=(SeCreTCODe)" http://stackoverflow.com/users/30/myProfile.html')
with open("myProfile.html") as f:
    for line in f:
        profile = profile + line
f.close()
p = re.compile('summarycount">(\d+)</div>') #Rep is found here
print p
m = p.search(profile)
print m
print m.group(1)
os.system("espeak \"Rep is at " + m.group(1) + " points\""
os.remove("myProfile.html")