How to turn display from none to block in Scrapy?

Question

I\'m trying to scrape data from a drop down menu(Here is the link). During inspecting to get the xpath, I realized that the display is none. So is there any way to scrape da

Answer 1

The data you want to scrape gets populated via Ajax call. So, you need to find out the url of the Ajax call. Once, you get that ,your work is easy.

Follow the steps below.

1. Open Chrome
2. Open the link
3. Open Developers console
4. Go to network tab
5. Now click on "Fits the following cars"
6. In the network tab ,see the call happening

In your case, it's a post request that happens over the fly.

Here is the pic of the call

Therefore, you need to find the url and the request parameters passed during the request.

You can see that the request parameters are as follows:

1. catentryId: 31426
2. techDocId: 33503

Now you got the url and data, it's just a matter of few lines of code.