读取大文件时Java StAX解析器出现问题

Question

I am trying to read a XML file using StAX parser which is having nearly 180k lines. Core logic looks for certain tags, attributes and stores in a data structure. For this type of large files StAX parser is taking lot of time. It is taking nearly 15 minutes without any core logic, just iterating over the while loop.

while (eventReader.hasNext()) { }

I tried SAX parser on the same file to just read the tags. It is very fast and completed in couple of seconds.

What would be the issue with StAX parser.? Please suggest any XML parser which is suitable for large files and perform well with respective of memory and space utilization. ?

Answer 1

Calling hasNext() will always return true unless you have reached the end of the input, and your code doesn't change position in the input because it never reads any data. You need to call next() in the loop, then eventually hasNext() will return false.

Incidentally 180k lines is not a large file by modern standards.

Answer 2

Stick with StAX parser as both SAX and Stax follows a Streaming programming model for parsing XML I ran the sample codes for both SAX and StAX here are the results

SAX Parser: Total Time Taken:10.73 ms max memory:1842688 allocated memory:125952 free memory:107293

StAX Parser: Total Time Taken:7.5 ms max memory:1842688 allocated memory:125952 free memory:120611

StAX is a PULL API, whereas SAX is a PUSH API means in case of StAx Parser a client application calls methods on an XML parsing library when it needs to interact with an XML infoset--that is, the client only gets (pulls) XML data when it explicitly asks for it.But in case of SAX parser,an XML parser sends (pushes) XML data to the client as the parser encounters elements in an XML infoset--that is, the parser sends the data whether or not the client is ready to use it at that time. StAX API can read as well as write XML documents. Using SAX API, an XML file can only be read.

StAX Code:

public static void main(String[] args) throws FileNotFoundException, XMLStreamException {
        XMLInputFactory xf=XMLInputFactory.newInstance();
        XMLStreamReader xsr=xf.createXMLStreamReader(new InputStreamReader(new FileInputStream("C:\\Users\\RNayyar\\Desktop\\Context\\processedFiles\\post.xml")));
        String startElement = null;
        String endElement  =null;
        String elementTxt = null;
        SimpleDateFormat dateFormat = new SimpleDateFormat("MM-dd-yyyy HH:mm:ss");

        while (xsr.hasNext()) {
            int e = xsr.next();
            if(e==XMLStreamConstants.START_ELEMENT){
                //System.out.println("StartElement Name :" + xsr.getLocalName());
                startElement = xsr.getLocalName();
            }
            if(e==XMLStreamConstants.END_ELEMENT){
                //System.out.println("EndElement Name :" + xsr.getLocalName());
                endElement = xsr.getLocalName();
                if(startElement.equalsIgnoreCase(endElement))
                System.out.println(" ElementName : "+ startElement + " ElementText : " + elementTxt);
            }
            if(e==XMLStreamConstants.CHARACTERS){
                //System.out.println("Element TextValue :" + xsr.getText());
                elementTxt = (xsr.getText().contains("\n")) ? "" : xsr.getText();
            }

        }
    }