How to find XML Elements via XPath in Python in a namespace-agnostic way?

Question

since I had this annoying issue for the 2nd time, I thought that asking would help.

Sometimes I have to get Elements from XML documents, but the ways to do this are

Answer 1

The *[local-name() = "elem"] syntax should work, but to make it easier you can create a function to simplify construction of the partial or full "wildcard namespace" XPath expressions.

I'm using python-lxml 2.2.4 on Ubuntu 10.04 and the script below works for me. You'll need to customize the behavior depending on how you want to specify the default namespaces for each element, plus handle any other XPath syntax you want to fold into the expression:

import lxml.etree

def xpath_ns(tree, expr):
    "Parse a simple expression and prepend namespace wildcards where unspecified."
    qual = lambda n: n if not n or ':' in n else '*[local-name() = "%s"]' % n
    expr = '/'.join(qual(n) for n in expr.split('/'))
    nsmap = dict((k, v) for k, v in tree.nsmap.items() if k)
    return tree.xpath(expr, namespaces=nsmap)

doc = '''<root xmlns="http://really-long-namespace.uri"
    xmlns:other="http://with-ambivalent.end/#">
    <other:elem/>
</root>'''

tree = lxml.etree.fromstring(doc)
print xpath_ns(tree, '/root')
print xpath_ns(tree, '/root/elem')
print xpath_ns(tree, '/root/other:elem')

Output:

[<Element {http://really-long-namespace.uri}root at 23099f0>]
[<Element {http://with-ambivalent.end/#}elem at 2309a48>]
[<Element {http://with-ambivalent.end/#}elem at 2309a48>]

Update: If you find out you do need to parse XPaths, you can check out projects like py-dom-xpath which is a pure Python implementation of (most of) XPath 1.0. In the least that will give you some idea of the complexity of parsing XPath.

Answer 2

First, about "what you want to do":

1. Unnamespaced: /root/elem -> no problem here I presume
2. Namespace-prefixes from document: /root/other:elem -> well, that's a bit of a problem, you cannot just use "namespace-prefixes from document". Even within one document:
   • namespaced elements do not necessarily even have a prefix
   • the same prefix isn't necessarily always mapped to the same namespace uri
   • the same namespace uri doesn't necessarily always have the same prefix

FYI: if you want to get to the prefix mappings in scope for a certain element, try elem.nsmap in lxml. Also, the iterparse and iterwalk methods in lxml.etree can be used to be "notified" of namespace declarations.