Scrapy or Selenium or Mechanize to scrape web data?

Question

I want to scrape some data from a website.

Basically, the website has some tabular display and shows around 50 records. For more records, the user has to click some butt

Answer 1

I would recommend you to go with a combination of Mechanize and ExecJS (https://github.com/sstephenson/execjs) to execute any javascript requests you might come across. I have used those two gems in combination for quite some time now and they do a great job.

You should choose this instead of Selenium, because it it will be a lot faster compared to having to render the entire page in a headless browser.

Answer 2

Definitely I'd choose Scrapy. If you can't handle javascript you can try with Scrapy + splash. Scrapy is by far the fastest tool for web scraping that I'm aware of. Good luck!