Scraping new ESPN site using xpath [Python]

依然范特西╮ 提交于 2019-12-03 09:08:01

The nature of the page is quite dynamic - there are asynchronous XHR requests, javascript logic involved. requests is not a browser and downloads only the initial HTML page and there are no span elements with class="time" in the HTML that requests gets.

One of the options to approach the problem would be to involve a real browser using selenium. Here is an example using PhantomJS headless browser:

>>> from selenium import webdriver
>>> 
>>> url = "http://scores.espn.go.com/nba/scoreboard?date=20150405"
>>> 
>>> driver = webdriver.PhantomJS()
>>> driver.get(url)
>>> 
>>> elements = driver.find_elements_by_css_selector("span.time")
>>> for element in elements:
...     print element.text
... 

1:00 PM ET
3:30 PM ET
6:00 PM ET
7:00 PM ET
7:30 PM ET
9:00 PM ET
9:30 PM ET 

Alternatively, you can look for the desired data in the data-data attribute of the div with id="scoreboard-page":

import json
from pprint import pprint

import lxml.html
import requests

response = requests.get('http://scores.espn.go.com/nba/scoreboard?date=20150405')
doc = lxml.html.fromstring(response.content)

data = doc.xpath("//div[@id='scoreboard-page']/@data-data")[0]
data = json.loads(data)

pprint(data)
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!