VTD XML（Java）VTDNavHuge将XPath结果写入文件

Question

I am experimenting with VTD XML because I frequently need to modify huge XML files (2-10GB or more).

I am try to write an XPath Query result back to a file. Writing huge files in VTD XML is not obvious to me though:

1. The method getBytes() is "not implemented" for XMLMemMappedBuffer (see https://jar-download.com/javaDoc/com.ximpleware/vtd-xml/2.13/com/ximpleware/extended/XMLMemMappedBuffer.html )

2. One of the authors (?) gives a code example in this thread (last post, 2010-04-21): https://sourceforge.net/p/vtd-xml/discussion/379067/thread/a2e03ede/

However, the example is outdated as

long la = vnh.getElementFragment();

returns an Array long[] (see https://jar-download.com/java-documentation-javadoc.php?a=vtd-xml&g=com.ximpleware&v=2.13 )

Adapting the relevant lines like this

long[] la = vnh.getElementFragment();
vnh.getXML().writeToFileOutputStream(new FileOutputStream("c:/text2.xml"), (int)la[0], (int)la[1]);

results in the following error:

Exception in thread "main" java.nio.channels.ClosedChannelException
    at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source)
    at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
    at com.ximpleware.extended.XMLMemMappedBuffer.writeToFileOutputStream(XMLMemMappedBuffer.java:104)
    at WriteXML.main(WriteXML.java:16)

Questions:

• Is this error due to any obvious mistake in the code?
• What tools would you use to handle huge XML files (~10GB) efficiently? (Does not have to be Java.)

My goal is to do simple transformations or split the xml and write back to file with great performance. Thanks!

Answer 1

无法回答您的第一个问题，但是至于第二个问题，如果您正在寻找不同的技术，那么流XSLT 3.0是一个值得探索的问题：如果不查看您的要求的更多细节，就无法确定它是否真正合适。

Answer 2

First of all, to process XML of huge size as you mentioned, I suggest that you load xml into memory using mem-map mode. And since vtd-xml doesn't alter the underlying byte format of xml, you can easily imagine saving a lot of back-and-forth encoding/decoding byte-moving operations and the performance advantage thereof.

As you have pointed out, XMLMemMappedBuffer getBytes is not implemented... this is to avoid excessive memory usage when the fragment is very large...

your work around is to use XMLMemMappedBuffer's writeToFileOutputStream() method to directly dump it to output. In other words, if you know the offset and length of the fragment... getBytes is often bypass-able.

Below is the signature document of that method.

public void writeToFileOutputStream(java.io.FileOutputStream ost, long os, long len) throws java.io.IOException write the segment (denoted by its offset and length) into an output file stream