How to continously scroll down a page until an element is located? Python selenium

Question

I\'m baffled by the load more button for the skills section in Linkedin. I receive this error in finding the xpath for the button: selenium.common.exceptions.NoSuchElemen

Answer 1

Try this

actions.move_to_element(loadmore_skills).build().perform()

Here is corresponding working Java Code

@Test
public void linkedInTest(){
    driver.findElement(By.name("session_key")).sendKeys(username);
    driver.findElement(By.name("session_password")).sendKeys(password);
    driver.findElement(By.xpath("//button[@class='btn__primary--large from__button--floating']")).click();

    driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk");

    WebElement loadmore_skills = driver.findElement(By.xpath("//button[@class='pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid']"));
    action.moveToElement(loadmore_skills).click().build().perform();
}

Answer 2

Try below code :

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains


driver = webdriver.Chrome(executable_path="C:\New folder\chromedriver.exe")

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()

driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()

buttonClick = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()

Output:

Answer 3

Here's the fully working code solving your problems as per you stated in the comments.

The answers suggested by others was actually scrolling down to bottom of the page and then also giving errors. Then i noticed that if you scroll drown to bottom then only the bottom section loads out, not all of those sections in between. (Dipak's answer isn't working for me also. Maybe this is a resolution problem for you, as well as for me : ) as stated by him in the chats)

Because what you want is present between the page & not in the bottom. So just the bottom section loads out always. So now we need to do something else.

What we want now, is to scroll down to only those sections which you want. And to make a custom scroll down i used driver.execute_script("scroll(0, 1600)"). I also removed all those unnecessary things from the code and kept it very simple and straight forward.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import time

driver = webdriver.Chrome(executable_path=r"C:\Users\intel\Downloads\Setups\chromedriver")

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("EMAIL")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("PASSWORD")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()

driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()

driver.execute_script("scroll(0, 1600)")
time.sleep(5)
buttonClick = driver.find_element_by_xpath("/html/body/div[6]/div[4]/div[3]/div/div/div/div/div[2]/main/div[2]/div[6]/div/section/div[2]/button/span[1]").click()

Also use --headless browser to load the tasks more quicker. And if possible, then use css_selectors other than XPATH's, cauz they are the slowest locators for scraping purpose.