Java Stax for Complex / Large XML

Question

I have an XML file that is 4.2 GB! Obviously parsing the entire DOM is not practical. I have been looking at SAX and STAX to accomplish parsing this gigantic XML file. However all the examples I've seen are simple. The XML file I am dealing with has nested on nested on nested. There are areas where it goes 10+ levels.

I found this tutorial but not sure if its a viable solution.

http://www.javacodegeeks.com/2013/05/parsing-xml-using-dom-sax-and-stax-parser-in-java.html (botton example using STAX)

I'm not really sure how to handle nested objects.

I have created Java objects to mimic the structure of the XML. Here are a few, too many to display.

Record.java

public class Record implements Serializable {

    String uid;
    StaticData staticData;
    DynamicData dynamicData;
}

Summary.java

public class Summary {

    EWUID ewuid;
    PubInfo pubInfo;
    Titles titles;
    Names names;
    DocTypes docTypes;
    Publishers publishers;
}

EWUID.java

public class EWUID {

    String collId;
    String edition;
}

PubInfo.java

public class PubInfo {

    String coverDate;
    String hasAbstract;
    String issue;
    String pubMonth;
    String pubType;
    String pubYear;
    String sortDate;
    String volume;
}

This is the code I've come up with so far.

public class TRWOSParser {

    XMLEventReader eventReader;
    XMLInputFactory inputFactory;
    InputStream inputStream;

    public TRWOSParser(String file) throws FileNotFoundException, XMLStreamException {
        inputFactory = XMLInputFactory.newInstance();
        inputStream = new FileInputStream(file);
        eventReader = inputFactory.createXMLEventReader(inputStream);
    }

    public void parse() throws XMLStreamException{

        while (eventReader.hasNext()) {
            XMLEvent event = eventReader.nextEvent();

            if (event.isStartElement()) {
                StartElement startElement = event.asStartElement();
                if (startElement.getName().getLocalPart().equals("record")) {
                    Record record = new Record();
                    Iterator<Attribute> attributes = startElement.getAttributes();
                    while (attributes.hasNext()) {
                        Attribute attribute = attributes.next();
                        if (attribute.getName().toString().equals("UID")) {
                            System.out.println("UID: " + attribute.getValue());
                        }
                    }
                }
            }
        }
    }
}

Update:

The data in the XML is licensed so I cannot show the full file. This is a very very small segment in which I have scrambled the data.

<?xml version="1.0" encoding="UTF-8"?>
<records>
    <REC>
        <UID>WOS:000310438600004</UID>
        <static_data>
            <summary>
                <EWUID>
                    <WUID coll_id="WOS" />
                    <edition value="WOS.SCI" />
                </EWUID>
                <pub_info coverdate="NOV 2012" has_abstract="N" issue="5" pubmonth="NOV" pubtype="Journal" pubyear="2012" sortdate="2012-11-01" vol="188">
                    <page begin="1662" end="1663" page_count="2">1662-1663</page>
                </pub_info>
                <titles count="6">
                    <title type="source">JOURNAL OF UROLOGY</title>
                    <title type="source_abbrev">J UROLOGY</title>
                    <title type="abbrev_iso">J. Urol.</title>
                    <title type="abbrev_11">J UROL</title>
                    <title type="abbrev_29">J UROL</title>
                    <title type="item">Something something</title>
                </titles>
                <names count="1">
                    <name addr_no="1 2 3" reprint="Y" role="author" seq_no="1">
                        <display_name>John Doe</display_name>
                        <full_name>John Doe</full_name>
                        <wos_standard>Doe, John</wos_standard>
                        <first_name>John</first_name>
                        <last_name>Doe</last_name>
                    </name>
                </names>
                <doctypes count="1">
                    <doctype>Editorial Material</doctype>
                </doctypes>
                <publishers>
                    <publisher>
                        <address_spec addr_no="1">
                            <full_address>360 PARK AVE SOUTH, NEW YORK, NY 10010-1710 USA</full_address>
                            <city>NEW YORK</city>
                        </address_spec>
                        <names count="1">
                            <name addr_no="1" role="publisher" seq_no="1">
                                <display_name>ELSEVIER SCIENCE INC</display_name>
                                <full_name>ELSEVIER SCIENCE INC</full_name>
                            </name>
                        </names>
                    </publisher>
                </publishers>
            </summary>
        </static_data>
    </REC>
</records>

Answer 1

A similar solution to lscoughlin's answer is to use DOM4J which has mechanims to deal with this scenario: http://dom4j.sourceforge.net/

In my opionin it is more straight forward and easier to follow. It might not support namespaces, though.

Answer 2

I'm making two assumptions 1) that there is an early level of repetition, and 2) that you can do something meaningful with a partial document.

Let's assume you can move some level of nesting in, and then handle the document multiple times, removing the nodes at the working level each time you "handle" the document. This means that only a single working subtree will be in memory at any given time.

Here's a working code snippet:

package bigparse;

import static javax.xml.stream.XMLStreamConstants.CHARACTERS;
import static javax.xml.stream.XMLStreamConstants.END_DOCUMENT;
import static javax.xml.stream.XMLStreamConstants.END_ELEMENT;
import static javax.xml.stream.XMLStreamConstants.START_DOCUMENT;
import static javax.xml.stream.XMLStreamConstants.START_ELEMENT;

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.StringWriter;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class BigParse {

    public static void main(String... args) {

        XMLInputFactory factory = XMLInputFactory.newInstance();
        DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();

        try {

            XMLStreamReader streamReader = factory.createXMLStreamReader(new FileReader("src/main/resources/test.xml"));
            DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();

            Document document = documentBuilder.newDocument();
            Element rootElement = null;
            Element currentElement = null;

            int branchLevel = 0;
            int maxBranchLevel = 1;

            while (streamReader.hasNext()) {
                int event = streamReader.next();
                switch (event) {
                case START_DOCUMENT:
                    continue;
                case START_ELEMENT:

                    if (branchLevel < maxBranchLevel) {
                        Element workingElement = readElementOnly(streamReader, document);

                        if (rootElement == null) {
                            document.appendChild(workingElement);
                            rootElement = document.getDocumentElement();
                            currentElement = rootElement;
                        } else {
                            currentElement.appendChild(workingElement);
                            currentElement = workingElement;
                        }

                        branchLevel++;
                    } else {

                        workingLoop(streamReader, document, currentElement);

                    }

                    continue;
                case CHARACTERS:
                    currentElement.setTextContent(streamReader.getText());
                    continue;

                case END_ELEMENT:
                    if (currentElement != rootElement) {
                        currentElement = (Element) currentElement.getParentNode();
                        branchLevel--;
                    }

                    continue;

                case END_DOCUMENT:
                    break;
                }

            }

        } catch (ParserConfigurationException
                | FileNotFoundException
                | XMLStreamException e) {
            throw new RuntimeException(e);
        }

    }

    private static Element readElementOnly(XMLStreamReader streamReader, Document document) {

        Element workingElement = document.createElement(streamReader.getLocalName());

        for (int attributeIndex = 0; attributeIndex < streamReader.getAttributeCount(); attributeIndex++) {
            workingElement.setAttribute(
                    streamReader.getAttributeLocalName(attributeIndex),
                    streamReader.getAttributeValue(attributeIndex));

        }
        return workingElement;
    }

    private static void workingLoop(final XMLStreamReader streamReader, final Document document, final Element fragmentRoot)
            throws XMLStreamException {

        Element startElement = readElementOnly(streamReader, document);
        fragmentRoot.appendChild(startElement);

        Element currentElement = startElement;

        while (streamReader.hasNext()) {
            int event = streamReader.next();
            switch (event) {
            case START_DOCUMENT:
                continue;
            case START_ELEMENT:

                Element workingElement = readElementOnly(streamReader, document);
                currentElement.appendChild(workingElement);
                currentElement = workingElement;

                continue;
            case CHARACTERS:
                currentElement.setTextContent(streamReader.getText());
                continue;

            case END_ELEMENT:
                if (currentElement != startElement) {
                    currentElement = (Element) currentElement.getParentNode();
                    continue;

                } else {

                    handleDocument(document, startElement);

                    fragmentRoot.removeChild(startElement);
                    startElement = null;
                    return;
                }

            }

        }

    }

    // THIS FUNCTION DOES SOMETHING MEANINFUL
    private static void handleDocument(Document document, Element startElement) {

        System.out.println(stringify(document));
    }

    private static String stringify(Document document) {

        try {
            Transformer transformer = TransformerFactory.newInstance().newTransformer();
            transformer.setOutputProperty(OutputKeys.INDENT, "yes");

            StreamResult result = new StreamResult(new StringWriter());
            DOMSource source = new DOMSource(document);

            transformer.transform(source, result);

            String xmlString = result.getWriter().toString();
            return xmlString;

        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

EDIT: I made an incredibly silly mistake. It's fixed now. It's working but imperfect -- should be enough to lead you in a useful direction.

Answer 3

Consider using an XSLT 3.0 streaming transformation of the form:

<xsl:template name="main">
  <xsl:stream href="bigInput.xml">
    <xsl:for-each select="copy-of(/records/REC)">
      <!-- process one record -->
    </xsl:for-each>
  </xsl:stream>
</xsl:template>

You can process this using Saxon-EE 9.6.

The "process one record" logic could use the Saxon SQL extension, or it could invoke an extension function: the context node will be a REC element with its contained tree, fully navigable within the subtree, but with no ability to navigate outside the REC element currently being processed.