问题
I've been tasked with fixing a rather irritating Heap out of memory issue. IBM offers a Cognos SDK that we use with Java, and we query all of the packages stored on a content store, which are returned in an xml format. Then we parse that xml and write it to a sql database. Profiling reveals that the worst memory issues are caused by Char[], which isn't very helpful (and the heaps are so large it's hard to profile), but does point towards the DOM parser.
We're talking 500-1500 xml files (well, technically, XML text streams) that are absurdly deeply nested and vary in size and occasionally in structure. Size varies from a few KB up to 30 MB in size, and the program will eat upwards of 8 GB of memory after about 300 packages. Programmer before me handled this by doing a manual System.gc call after every xml parse, which I wish to move away from (and it also doesn't actually solve the issue, just makes it viable on the smallest, 500 package server).
I tried to use JAXB, but it has an odd structure that made it very difficult to use here (it has some "folder or querySubject" thing going on). I tried STAX for several hours last week, but wasn't able to quite get working, same for WoodStox. I couldn't really find examples or tutorials on doing this for either. JDOM was what I examined next (as I've read that it has significantly better memory handling than pure DOM), but I can't figure out how to get it to parse quite as deeply as DOM. Current DOM parsing :
is = new ByteArrayInputStream(xml.getBytes("UTF-8"));
xmlDoc = builder.parse(is);
is.close();
String _path, datatype, regularAggregate, description, formula;
String table, tableLoc;
NodeList elements = xmlDoc.getElementsByTagName("*");
for (int j = 0; j < elements.getLength(); j++) {
Element element = (Element) elements.item(j);
String nodeName = element.getNodeName();
if (nodeName=="queryItem" || nodeName=="measure"||
nodeName=="calculation" || nodeName=="filter") {
if (element.hasAttribute("_path")) {
path = element.getAttribute("_path"));
}
and so on for each attribute
My JDOM attempt. Currently, it only prints the root element, and I've yet to be able to go deeper than the first child layer :
SAXBuilder saxBuilder = new SAXBuilder();
Document document = saxBuilder.build(inputFile);
System.out.println("Root element :" + document.getRootElement().getName());
Element root = document.getRootElement();
List<Element> rList = root.getChildren("folder");
if (rList!= null) {
for (Element node : rList) {
List<Element> elements = node.getChildren("queryItem");
if (elements!=null) {
for (Element a:elements) {
System.out.println(a.getAttribute("_path"));
}
elements.size();
rList.removeAll(elements);
}
}
Generated xsd structure of a random package:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
<xs:element name="ResponseRoot">
<xs:complexType>
<xs:sequence>
<xs:element ref="folder"/>
<xs:element ref="package"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="package">
<xs:complexType>
<xs:attribute name="description" use="required"/>
<xs:attribute name="name" use="required"/>
<xs:attribute name="screenTip" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="folder">
<xs:complexType>
<xs:sequence>
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element ref="folder"/>
<xs:element ref="querySubject"/>
</xs:choice>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="filter"/>
</xs:sequence>
<xs:attribute name="_path" use="required"/>
<xs:attribute name="_ref" use="required"/>
<xs:attribute name="description" use="required"/>
<xs:attribute name="isNamespace" use="required" type="xs:integer"/>
<xs:attribute name="name" use="required"/>
<xs:attribute name="screenTip" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="querySubject">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="queryItem"/>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="queryItemFolder"/>
</xs:sequence>
<xs:attribute name="_path" use="required"/>
<xs:attribute name="_ref" use="required"/>
<xs:attribute name="description" use="required"/>
<xs:attribute name="name" use="required"/>
<xs:attribute name="screenTip" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="filter">
<xs:complexType>
<xs:attribute name="_path" use="required"/>
<xs:attribute name="_ref" use="required"/>
<xs:attribute name="description" use="required"/>
<xs:attribute name="expression" use="required"/>
<xs:attribute name="name" use="required"/>
<xs:attribute name="screenTip" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="queryItem">
<xs:complexType>
<xs:attribute name="_path" use="required"/>
<xs:attribute name="_ref" use="required"/>
<xs:attribute name="currency" use="required"/>
<xs:attribute name="datatype" use="required" type="xs:NCName"/>
<xs:attribute name="description" use="required"/>
<xs:attribute name="displayType" use="required" type="xs:NCName"/>
<xs:attribute name="expression" use="required"/>
<xs:attribute name="name" use="required"/>
<xs:attribute name="promptCascadeOnRef" use="required"/>
<xs:attribute name="promptDisplayItemRef" use="required"/>
<xs:attribute name="promptFilterItemRef" use="required"/>
<xs:attribute name="promptType" use="required" type="xs:NCName"/>
<xs:attribute name="regularAggregate" use="required" type="xs:NCName"/>
<xs:attribute name="screenTip" use="required"/>
<xs:attribute name="unSortable" use="required" type="xs:integer"/>
<xs:attribute name="usage" use="required" type="xs:NCName"/>
</xs:complexType>
</xs:element>
xs:element name="queryItemFolder">
<xs:complexType>
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element ref="queryItem"/>
<xs:element ref="queryItemFolder"/>
</xs:choice>
<xs:attribute name="_path" use="required"/>
<xs:attribute name="_ref" use="required"/>
<xs:attribute name="description" use="required"/>
<xs:attribute name="name" use="required"/>
<xs:attribute name="screenTip" use="required"/>
</xs:complexType>
</xs:element>
</xs:schema>
回答1:
For nested structures, it's easiest to manage if you create a method for each element type.
Example
public static void main(String[] args) throws Exception {
String xml = "<root>" +
"<folder name=\"A\">" +
"<folder name=\"B\">" +
"<book name=\"Learn Java\">" +
"<chapter name=\"Hello, World!\"/>" +
"<chapter name=\"Variables and Types\"/>" +
"</book>" +
"</folder>" +
"</folder>" +
"</root>";
XMLInputFactory factory = XMLInputFactory.newFactory();
XMLStreamReader reader = factory.createXMLStreamReader(new StringReader(xml));
try {
reader.nextTag(); // Position on root element
String tagName = reader.getLocalName();
if (! tagName.equals("root"))
throw new XMLStreamException("Expected <root> element, found: " + tagName, reader.getLocation());
parseRoot(reader);
} finally {
reader.close();
}
}
private static void parseRoot(XMLStreamReader reader) throws XMLStreamException {
while (reader.nextTag() != XMLStreamConstants.END_ELEMENT) {
String tagName = reader.getLocalName();
if (tagName.equals("folder")) {
parseFolder(reader, Collections.emptyList());
} else {
throw new XMLStreamException("Expected <folder> element, found: " + tagName, reader.getLocation());
}
}
}
private static void parseFolder(XMLStreamReader reader, List<String> parentPaths) throws XMLStreamException {
String folderName = reader.getAttributeValue(null, "name");
if (folderName == null)
throw new XMLStreamException("Missing 'name' attribute on <folder> element", reader.getLocation());
List<String> folderPath = new ArrayList<>(parentPaths.size() + 1);
folderPath.addAll(parentPaths);
folderPath.add(folderName);
while (reader.nextTag() != XMLStreamConstants.END_ELEMENT) {
String tagName = reader.getLocalName();
if (tagName.equals("folder")) {
parseFolder(reader, folderPath);
} else if (tagName.equals("book")) {
parseBook(reader, folderPath);
} else {
throw new XMLStreamException("Expected <folder> or <book> element, found: " + tagName, reader.getLocation());
}
}
}
private static void parseBook(XMLStreamReader reader, List<String> folderPath) throws XMLStreamException {
String bookName = reader.getAttributeValue(null, "name");
if (bookName == null)
throw new XMLStreamException("Missing 'name' attribute on <book> element", reader.getLocation());
while (reader.nextTag() != XMLStreamConstants.END_ELEMENT) {
String tagName = reader.getLocalName();
if (tagName.equals("chapter")) {
parseChapter(reader, folderPath, bookName);
} else {
throw new XMLStreamException("Expected <chapter> element, found: " + tagName, reader.getLocation());
}
}
}
private static void parseChapter(XMLStreamReader reader, List<String> folderPath, String bookName) throws XMLStreamException {
String chapterName = reader.getAttributeValue(null, "name");
if (chapterName == null)
throw new XMLStreamException("Missing 'name' attribute on <chapter> element", reader.getLocation());
if (! reader.getElementText().isEmpty())
throw new XMLStreamException("<chapter> element must be empty", reader.getLocation());
System.out.println("Found:");
System.out.println(" Folder: " + folderPath);
System.out.println(" Book: " + bookName);
System.out.println(" Chapter: " + chapterName);
}
Output
Found:
Folder: [A, B]
Book: Learn Java
Chapter: Hello, World!
Found:
Folder: [A, B]
Book: Learn Java
Chapter: Variables and Types
来源：https://stackoverflow.com/questions/52614082/parsing-highly-nested-xml-without-dom-in-java