requests-html and infinite scrolling

半城伤御伤魂 提交于 2021-02-10 05:50:55

问题


I'm checking a python library: requests-html. Looks interesting, easy and clear scraping. However, I'm not sure how to render a page with infinite scrolling.

From their documentation I understand that I should render a page with special attribute (scrolldown). I'm trying but I do not know how exactly. I know how to use selenium to handle infinite scroll, but I wonder whether it is possible with requests-html.

from requests_html import  HTML, HTMLSession

page1 = session.get(url1)
page1.html.render( scrolldown=5,sleep=3)
html = HTML(html=page1.text)
noticeName = html.find('h2.noticeName')
for element in noticeName:
    print(element.text)

It finds 10 elements from 13. 10 is visible without scrolling (and loading new content because of infinite scroll).


回答1:


scrolldown=5 means scroll 5 pixel down, is your monitor that small?? or vm height that small?? now give it a bigger value like height of the screen with sleep or 2000 or 5000 without sleep

And it will not give you uniquely next elements, it will give you exactly all elements from the starting.

I will add some sample code soon.



来源:https://stackoverflow.com/questions/56586016/requests-html-and-infinite-scrolling

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!