I\'m using RSelenium to automatically scroll down a social media website and save posts. Sometimes I get to the bottom of the webpage and no more posts can be loaded as no m
Stumbled across a way to do this in Python here and modified it to work in R. Below is a now-working update of the original code I posted above.
# Open webpage
library(RSelenium)
rD = rsDriver(browser = "firefox")
remDr = rD[["client"]]
url = "https://stocktwits.com/symbol/NZDCHF"
remDr$navigate(url)
# Keep scrolling down page, loading new content each time.
last_height = 0 #
repeat {
remDr$executeScript("window.scrollTo(0,document.body.scrollHeight);")
Sys.sleep(3) #delay by 3sec to give chance to load.
# Updated if statement which breaks if we can't scroll further
new_height = remDr$executeScript("return document.body.scrollHeight")
if(unlist(last_height) == unlist(new_height)) {
break
} else {
last_height = new_height
}
}