How do I crawl an infinite-scrolling page?

Question

I\'m trying to build something that crawls the content from a page with infinite scroll. However, I can\'t get the stuff from below the first \'break\'. How do I do this?

Answer 1

An ajax request is no different from any other request. You simply make the request, parse the result, and there you have your data.

It can take some experience if you haven't done it before but it sounds like a good learning experience.