Requests won't get the text from web page?

Question

I am trying to get the value of VIX from a webpage.

The code I am using:

raw_page = requests.get(\"https://www.nseindia.com/live_market/dynaContent/live         


        

Answer 1

When you open the page in a web browser, the text (e.g., 15.785) is inserted into the span element by the getIndiaVixData.js script.

When you get the page using requests in Python, only the HTML code is retrieved and no JavaScript processing is done. So, the span element stays empty.

It is impossible to get that data by solely parsing the HTML code of the page using requests.

Answer 2

The data you're looking for, is not available in the page source. And requests.get(...) gets you only the page source without the elements that are dynamically added through JavaScript. But, you can still get it using requests module.

In the Network tab, inside the developer tools, you can see a file named VixDetails.json. A request is being sent to https://www.nseindia.com/live_market/dynaContent/live_watch/VixDetails.json, which returns the data in the form of JSON.

You can access it using the built-in .json() function of the requests module.

r = requests.get('https://www.nseindia.com/live_market/dynaContent/live_watch/VixDetails.json')
data = r.json()
vix_price = data['currentVixSnapShot'][0]['CURRENT_PRICE']
print(vix_price)
# 15.7000