How to scrape project urls from indiegogo using BeautifulSoup?

江枫思渺然 提交于 2021-02-11 15:40:46

问题


I am trying to scrape project URLs from Indiegogo, but I had no success after hours. I can not scrape them either using XPath or Beautifulsoup. The output of the following code does not contain the information I want:

soup.find_all("div")

Also, Beutifulsoup did not work:

import requests
from bs4 import BeautifulSoup

url = 'https://www.indiegogo.com/explore/all?project_type=campaign&project_timing=ending_soon&sort=trending'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')

project_name_list = soup.find(class_='exploreDetail-campaigns row')

project_name_list_items = project_name_list.find_all('a')
print(project_name_list_items)

for project_name in project_name_list_items:
    links = project_name.get('href')
    print(links)

Actually, the following command returns nothing.

soup.find(class_='exploreDetail-campaigns row')

How should I know if a website is in javascript format or another format?

来源:https://stackoverflow.com/questions/61536072/how-to-scrape-project-urls-from-indiegogo-using-beautifulsoup

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!