I scrapped some html via xpath, that I then converted into an etree. Something similar to this:
text1 link text2
<
Use element.xpath("string()")
or lxml.etree.tostring(element, method="text")
- see the documentation.
As a public service to people out there who may be as lazy as I am. Here's some code from above that you can run.
from lxml import etree
def get_text1(node):
result = node.text or ""
for child in node:
if child.tail is not None:
result += child.tail
return result
def get_text2(node):
return ((node.text or '') +
''.join(map(get_text2, node)) +
(node.tail or ''))
def get_text3(node):
return (node.text or "") + "".join(
[etree.tostring(child) for child in node.iterchildren()])
root = etree.fromstring(u"<td> text1 <a> link </a> text2 </td>")
print root.xpath("text()")
print get_text1(root)
print get_text2(root)
print root.xpath("string()")
print etree.tostring(root, method = "text")
print etree.tostring(root, method = "xml")
print get_text3(root)
Output is:
snowy:rpg$ python test.py
[' text1 ', ' text2 ']
text1 text2
text1 link text2
text1 link text2
text1 link text2
<td> text1 <a> link </a> text2 </td>
text1 <a> link </a> text2
def get_text_recursive(node):
return (node.text or '') + ''.join(map(get_text_recursive, node)) + (node.tail or '')
looks like an lxml bug to me, but according to design if you read the documentation. I've solved it like this:
def node_text(node):
if node.text:
result = node.text
else:
result = ''
for child in node:
if child.tail is not None:
result += child.tail
return result
element.xpath('normalize-space()') also works.
If the element
is equal to <td>
. You can do the following.
element.xpath('.//text()')
It will give you a list of all text elements from self
(the meaning of the dot). //
means that it will take all elements and finally text()
is the function to extract text.