I have Python code for parsing an XML file as detailed here. I understand that XML files are notorious for hogging system resources when manipulated in memory. My solution works
You can parse your big XML file incrementally:
from xml.etree.cElementTree import iterparse
# get an iterable and turn it into an iterator
context = iter(iterparse("path/to/big.xml", events=("start", "end")))
# get the root element
event, root = next(context)
assert event == "start"
for event, elem in context:
if event == "end" and elem.tag == "book":
# ... process book elements ...
root.clear()
You can use elementtree.iterparse and discard each book tag after it is processed.