'NoneType' object has no attribute 'text' in BeautifulSoup

Question

I am trying to scrape Google results when I search \"What is 2+2\", but the following code is returning \'NoneType\' object has no attribute \'text\'.

Answer 1

Next time use the query string exactly as it is.

import requests
from bs4 import BeautifulSoup
search="2%2B2"
link="https://www.google.com/search?q="+search
headers={'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'}
source=requests.get(link,headers=headers).text
soup=BeautifulSoup(source,"html.parser")
answer=soup.find('span',id="cwos")
print(answer.text)

Output:

 4  

Visit these urls - they do not return the same result

https://www.google.com/search?q=What+is+2+2

https://www.google.com/search?q=2%2B2

https://www.google.com/search?q=2+2

Answer 2

A big gotcha when parsing websites is that the source code can look very different in your browser when compared to what requests sees. The difference is javascript, which can hugely modify the DOM in a javascript capable browser.

I'd suggest 3 options:

1. use requests to get the page, and then examine it closely - does that tag exist when the page is retrieved by a non-js enabled agent?
2. use https://www.seleniumhq.org/ as your agent - it's essentially a fully featured browser that you can control programatically, inc w/ python.
3. use google's search API instead of trying to scrape the html

Answer 3

When you run the code, you might encounter an AttributeError:

shell: AttributeError: 'NoneType' object has no attribute 'text'

If that’s the case, then take a step back and inspect your previous results. Were there any items with a value of None? You might have noticed that the structure of the page is not entirely uniform. There could be an advertisement in there that displays in a different way than the normal job postings, which may return different results

Reference:https://realpython.com/beautiful-soup-web-scraper-python/#extract-text-from-html-elements.