Tracking with JavaScript if Ajax request is going on in a webpage or Intercept XMLHttpRequest through Selenium Web driver

Question

I am using Selenium WebDriver for crawling a web site(only for example, I will be crawling other web sites too!) which has infinite scroll.

Problem statement:

Answer 1

Approach 1:

Your approach is good, just a few changes would do the trick:

Step 1: Improve this step to call the toBottom function at regular interval using window.setInterval. At (c >= totalcount) call window.clearInterval

Setp 2: Instead of checking the page is yet scrollable, check if (c >= totalcount). And this condition every 200ms until (c >= totalcount) returns true.

FYI: If the Step 1 doesn't work in all the browsers then probably, you can refer to line 5210 of Tata-Nano-Reviews-925076578.js and call this with cvariable checking.

Approach 2:

Go to jQuery API and type "ajax". You can find some callback handlers which could be used for ajax requests.

Probably, set a variable before the request is been sent and after it is been received appropriately.

And in between use your original method of scrolling to bottom at regular interval, unless you can no more scroll. At this point clear the interval variable.

Now, regularly check if that interval variable is null or not. Null would mean that you have reached the bottom.