How to get content of a javascript/ajax -loaded div on a site?

Question

I have a PHP-script that loads page-content from another website by using CURL and simple_html_dom PHP library. This works great. If I echo out the HTML returned I can see the d

Answer 1

Unfortunately Javascript is run client-side, in a browser, so unless the page is loaded in a web browser there is no simple way to do it.

The only way I can think of, is having a browser running in a server’s background, reloading and saving the generated page automatically in a file which will be available for a PHP script to fetch. Well... I don’t know about anyone who has implemented such an idea.

Better try to get the URL where the div is being populated from. If the div contents are generated through AJAX for example, maybe if you fetch the data-origin URL with cURL, the data will be available for you as well.

Answer 2

Yes, it is possible.

What you need to do is the following:

1. Create a CURL call to that webpage in order to retrieve any parameter used in the Ajax call that loads the content, which you are looking for.
2. Create another CURL call to the file called by that webpage Javascript using the parameters that you have gotten using step number 1.

ex. Say you want to get the content of http://www.domain.com/page.html and this page.html retrieves some other data using Ajax, say $("#div").load("http://www.domain.com/ajax/data.php?time=48484&c=487387").

What you will do is to make a CURL request to page.html first, and get the full URL of the Ajax call using preg_match() PHP function or any equivalent function in any other language. After that, create another CURL request to that URL - http://www.domain.com/ajax/data.php?time=48484&c=487387 - and get its content.

You're all set!

Answer 3

For this kind of screen scraping you could try phpQuery or Snoopy.

phpQuery has a web browser plugin and scoopy claims to simulate one

Answer 4

Yes its piece of cake if you are interested only in that particular html which is returned by ajax.

1. Gather information like url, parameters and request type (post/get) from that ajax request.
2. Generate the same request from your php/curl code and you got it.
3. And hope that server logic will not check who sent the request.

Answer 5

you can always bind to the event that is fired when the xhr returns data to the browser and do your operations there.

 var xhReq = createXMLHttpRequest();
 xhReq.open("GET", "ur_php_url.php");
 xhReq.onreadystatechange = onResponse;
 xhReq.send(null);

 function onResponse()
 {
 // do the necessary
 }