Casperjs scraping dynamic content

给你一囗甜甜゛ 提交于 2019-11-29 17:13:50

I've looked to the page. It has such a behvior that it doesn't load the middle images when you jump to the end.

When the page is loaded the first couple of rows are completely loaded and some more are not completely loaded (image missing denoted by '.loading-msg' element). When you jump to the end with this.scrollToBottom(); there is no continous scroll. It jumps to the end and the page JavaScript doesn't detect that the middle images were in the viewport, however briefly. The page goes on to load the next rows, but not the missing images of the jumped over rows.

You have to reduce the distance of the jump in both of your snippets.

The first one can be changed like this:

var pos = 0, 
    height = casper.page.viewportSize.height;
casper.waitFor(function() {
    this.scrollTo(0, pos * height);
    return !this.exists('.loading-msg');
}, function() { // do stuff }, 20000);

The second one might work by changing

this.page.scrollPosition = { top: this.page.scrollPosition["top"] + 4000, left: 0 };

to

var height = casper.page.viewportSize.height;
this.page.scrollPosition = { top: this.page.scrollPosition.top + height, left: 0 };
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!