Python Selenium: I am getting element is not attached to the page document error

Question

I am trying to build a scraper using selenium package for python but I am getting this error:

Message: stale element reference: element is not attached to the pag         


        

Answer 1

It seems synchronization issue while iterating list of links. Induce WebDriverWait() and wait for visibility_of_all_elements_located() and then iterate and store in the list.

Use try..except block to handle.

Code:

titles=[]
for link in links:
  driver.get(link)
  try:
      data=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.XPATH,'//a[@class = "question-hyperlink"]')))
      for d in data:
        titles.append(d.text)
  except:
      print("element not found")
      continue

You need to import below libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By