用Java读取BIG XML文件的一小部分的有效方法

Question

We have a new requirement:

There are some BIG xml files keep coming into our system and we will need to process them immediately and quickly using Java. The file is huge but the required information for our processing is inside a element which is very small. ... ...

What is the best way to extract this small portion of the data from the huge file before we start processing. If we try to load the entire file, we will get out of memory error immediately due to size. What is the efficient way in Java that I can use to get the ..data..data..data.. data element without loading or reading the file line by line. Is there any SAX Parser that I can use to get this done?

Thank you

Answer 1

The SAX parsers are event based and are much faster because they do what you need: they don't read the xml document entirely. There is a SAXParser available in the Java distributions.

Answer 2

I had to parse huge files in a previous project (1G-2G) and didn't want to deal with using SAX. I find SAX too low-level in some instances and like keepings a traversal approach in most cases.

I have used the VTD library http://vtd-xml.sourceforge.net/ . It's an EXTREMELY fast library that uses pointers to navigate through the document.

Answer 3

Well, if you want to read a part of a file, you will need to read each line of the file to be able to identify the part of the file of interest and then extract what you need.

If you only need a small portion of the incoming XML, you can either use SAX, or if you need to read only specific elements or attributes, you could use XPath, which would be a lot simpler to implement.

Java comes with a built-in SAXParser implementation as well as an XPath implementation. Find the javadocs for SAXParser here and for XPath here .

Answer 4

StAX是基于漫游数据的另一种选择，如SAX，但是通过“拉动”您想要的而不是“推送”给您来处理数据的更友好的方法（IMO）。