'NoneType' object has no attribute 'text' in BeautifulSoup

前端 未结 3 1015
名媛妹妹
名媛妹妹 2021-01-13 03:07

I am trying to scrape Google results when I search \"What is 2+2\", but the following code is returning \'NoneType\' object has no attribute \'text\'.

3条回答
  •  栀梦
    栀梦 (楼主)
    2021-01-13 03:21

    A big gotcha when parsing websites is that the source code can look very different in your browser when compared to what requests sees. The difference is javascript, which can hugely modify the DOM in a javascript capable browser.

    I'd suggest 3 options:

    1. use requests to get the page, and then examine it closely - does that tag exist when the page is retrieved by a non-js enabled agent?
    2. use https://www.seleniumhq.org/ as your agent - it's essentially a fully featured browser that you can control programatically, inc w/ python.
    3. use google's search API instead of trying to scrape the html

自定义标题
段落格式
字体
字号
代码语言
提交回复
热议问题