Can scrapy be used to scrape dynamic content from websites that are using AJAX?

Question

I have recently been learning Python and am dipping my hand into building a web-scraper. It\'s nothing fancy at all; its only purpose is to get the data off of a betting we

Answer 1

yes, Scrapy can scrap dynamic websites, website that are rendered through javaScript.

There are Two approaches to scrapy these kind of websites.

First,

you can use splash to render Javascript code and then parse the rendered HTML. you can find the doc and project here Scrapy splash, git

Second,

As everyone is stating, by monitoring the network calls, yes, you can find the api call that fetch the data and mock that call in your scrapy spider might help you to get desired data.