Extracting node values using XPath

Question

There is a section of amazon.com from which I want to extract the data (node value only, not the link) for each item.

The value I\'m looking for is inside and <

Answer 1

If you need to grap the categories names:

// Suppress invalid markup warnings
libxml_use_internal_errors(true);

// Create SimpleXML object
$doc = new DOMDocument();
$doc->strictErrorChecking = false;
$doc->loadHTML($html); // $html - string fetched by CURL 
$xml = simplexml_import_dom($doc);

// Find a category nodes
$categories = $xml->xpath("//span[@class='refinementLink']");

EDIT. Using DOMDocument

$doc = new DOMDocument();
$doc->strictErrorChecking = false;
$doc->loadHTML($html);

$xpath = new DOMXPath($doc);

// Select the parent node
$categories = $xpath->query("//span[@class='refinementLink']/..");

foreach ($categories as $category) {
    echo '<pre>';
    echo $category->childNodes->item(1)->firstChild->nodeValue; 
    echo $category->childNodes->item(2)->firstChild->nodeValue;
    echo '</pre>';
    // Crafts, Hobbies & Home (19)
}

Answer 2

The following expression should work:

//*[@id='ref_1000']/li/a/span[@class='narrowValue']

For better performance you could provide a direct path to the start of this expression, but the one provided is more flexible (given that you probably need this to work across multiple pages).

Keep in mind, also, that your HTML parser might generate a different result tree than the one produced by Firebug (where I tested). Here's an even more flexible solution:

//*[@id='ref_1000']//span[@class='narrowValue']

Flexibility comes with potential performance (and accuracy) costs, but it's often the only choice when dealing with tag soup.

Answer 3

I'd highly recommend you checkout the phpQuery library. It's essentially the jQuery selectors engine for PHP, so to get at the text you're wanting you could do something like:

foreach (pq('span.refinementLink') as $p) {
  print $p->text() . "\n";
}

That should output something like:

Crafts, Hobbies & Home
Health, Fitness & Dieting
Cookbooks, Food & Wine

It's by far the easiest screen scraping, DOM parsing thing I know of for PHP.