How can I parse a dynamic page using Python? [duplicate]

Answer 1

Use phantomjs to open the page. Save it as a local file using phantomjs File System Module Api. Later use this local file handle to create BeautifulSoup object and then parse the page. See http://www.kochi-coders.com/2014/05/06/scraping-a-javascript-enabled-web-page-using-beautiful-soup-and-phantomjs/

Answer 2

Load the page in a real browser (headless like PhantomJS is also an option) automated by selenium, wait for the desired contents to appear, get the .page_source and pass it to BeautifulSoup:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.PhantomJS()
driver.get("your url here")

# waiting for the page to load - TODO: change
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.ID, "content")))

data = driver.page_source
driver.close()

soup = BeautifulSoup(data, "html.parser")