Selenium Chrome Driver Limitations Web Scraping at Scale

问题

I'm planning to use Selenium Chrome Driver for my project which will be used to do web scraping to multiple public websites (something like kayak or skyscanner). So there will be a REST GET endpoint where my backend would launch headless Chrome to scrape multiple websites, and eventually return a manipulated JSON.

I want to know how scalable is Chrome Driver as it sounds like a headless Chrome instance needs to be launched whenever a request comes in.

Updated: Question using Google Chrome Headless

回答1:

Please find the pros and cons of phantom js which I noticed during implementation .Hope this helps.

Cons:

1)It will fail to recognize the browser elements like id,xpath,csselector
when compared to chrome driver.
2)If you have login mechanism ,redirects won't work as you expect when compared to chrome driver.
3)You need to manually implement the custom logic for screen shots for the test failures if you need it.
4)If you want to switch between multiple drivers like chrome,html etc then it is very difficult

Pros:

1)Test case execution is faster when compared to chrome driver
2)No browser is required it will run without GUI. 
3)No much configurations are needed when compared to chromedriver.

You can go with html driver also which is quite faster then phantom but even it has its own limitations that you need take care of before implementation.

回答2:

I am not sure that you really need to use PhantomJS.

Chrome implemented "headless" mode couple of months ago. "Headless Chrome" does the same job that PhantomJS, and does it better. I heard that PhantomJS authors even said that they will not support it anymore.

You can enable headless mode in Selenide with just on line: