Web scraping, screen scraping, data mining tips? [closed]

Answer 1

I've found JSoup really good for HTML parsing.

For more pointers check this article out: How to write a multi-threaded webcrawler

Answer 2

I used Bixo for extracting the hyperlinks and images doing depth search,. It built over hadoop and cascading so there is a learning curve but the example provided is good enough to config the changes ...

Answer 3

Try using Web-Harvest project.

Answer 4

Checkout JSR-237 for Work Management, which is a cool idea when going multithreaded.

As for scraping, there are several alternatives. If ease of use is most important, I'd advise you to HTMLUnit. Beyond that, you must roll your own