BeautifulSoup returning none when element definitely exists

Question

I\'m new to web scraping and have been using BeautifulSoup to scrape daily mortgage rates. However, a lot of the servicer sites that I am trying to scrape return \'none\' or

Answer 1

If you check the page source (for example via view-source: in Chrome or Firefox, or by writing your html string to a file) you'll see that the element you are looking for is not there. In fact, the rates are loaded dynamically:

You can follow the script URL and you'll see that the response is something like the following:

document.write('
2.75%
')

This response is probably regular enough to be able to use regexes on it.