unable to scrape content from a website

Question

I am trying to scrap some content from a website but the code below is not working(not showing any output). here is the code

$url=\"some url\";
$otherHeader         


        

Answer 1

It seems that the problem is somehow related to XPath and namespaces. Php manual revealed an interesting user comment

If you've registered your namespaces, loaded your XHTML, etc., into your XPath's DOMDocument object and still can't get it to work, check to make sure you haven't used the DOMDocument's loadHTML() or loadHTMLFile() function. For XHTML always use the XML versions, otherwise your XPath will never, ever work.

Your code uses loadHTML()

$content=getXHTML($content);  //this is a tidy function to convert bad html to xhtml 
$page->loadHTML($content);    // its okay till here when i echo $page->saveHTML the page is displayed

HTML is not namespace aware so loadHTML() might not set the namespaces on the elements of the document object even though the original document (or the XHTML outputted by Tidy) had them.

Because you use Tidy to convert the document to XHTML, I guess you could safely use loadXML() without running into parsing errors. Note that it will require that the input is well-formed XML. Also it might not be aware of HTML predefined entities like   and if that is the case, it can't replace the entities with their correct character values. If such problem arises, try setting different options for loadXML().

Answer 2

(Try the following both in combination with and separately from the other answers, as they are other possible caveats.)

If your XPath isn't working, try applying just parts of it to make sure you are indeed following the right path. So do something like:

$path1="//body";
$item1 = $xpath->query($path1);

foreach ($item1 as $t) {
    // to see the full XML of the returned node, as the nodeValue may be empty
    echo $t->ownerDocument->saveXML($t); 
}

Then keep increasing your XPath to the location you want.

Also, if you find that nodeValue and textContent of your nodes are empty, you should make sure that you are loading into the DOMDocument with the correct encoding (e.g. if the cURL response returns UTF-8, you'll need to pass 'UTF-8' as the second parameter when constructing your DOMDOcument).

Answer 3

Yes, you are missing something very basic: It's XHTML, so you must register (and use!) the proper namespace before you can expect to get results.

$xpath->registerNamespace('x', 'http://www.w3.org/1999/xhtml');

$path1="//x:body/x:table[4]/x:tbody/x:tr[3]/x:td[4]";
$path2="//x:body/x:table[4]/x:tbody/x:tr[1]/x:td[4]";

$item1=$xpath->query($path1);
$item2=$xpath->query($path2);

Answer 4

I have heard that FireFox adds a tbody element if such isn't present.

In addition to or independently of @Tomalak's advice, try the XPath expressions with the /tbody location step removed.

Also, use another tool as the XPath Visualizer to construct correct XPath expressions and see immediately what they are selecting.

Answer 5

This question reminds me that a lot of times the solution to a problem lies in simplicity and not complications. i was trying namespaces,error corrections,etc but the solution just demanded close inspection of the code. the problem with my code was the order of loadHTML() and xpath initialization. initially the order was

$xpath=new DOMXPath($page);
$page->loadHTML($content);

by doing this i was actually initializing xapth on an empty document. now reversing the order by first loading the dom with the html and then initializing the xpath i was able to get the desired results. Also as suggested that by removing the tbodyelement from xpath as firefox automatically inserts it. so the correct xpath should be

$path1="//body/table[4]/tr[3]/td[4]";
$path2="//body/table[4]/tr[1]/td[4]";

thanks to everyone for their suggestions and bearing this.