创建数百万条记录的XML

Question

I have to create a single large XML file which will have more than 60 milllion records. I need a fast way to do this operation as it will be repeated on regular basis.

I have data at two places

1. In Database in two different tables.
2. On File System In two pipe separated files.

At present, I am trying to use StaX parser to create XML file in JAVA.

Here I need to know whether I should use database / text file to read data and create XML file.

Which one will be faster option ?
Also, is there any specific API which handles creation / conversion for such large data to XML in optimized way ?

I am open to File processing using linux commands as well provided it is a faster option.

UPDATE On Comments

• How fast can you get XML element info from two separate files vs DB Query CSV dump? Whatever is available in database is available in PIPE Separated Files. And How FAST can we get / create XML element info is exactly the intent of the question here. This is exactly what will impact overall speed of operation.

• Mapping Complication : Mapping is 1 to many. 1 records in first file maps to approx than 8000 records in second text file / table.

UPDATE on Comment - > Use large output buffers.

Understood. Will read data from database / file in larger chunks and flush it in once I have processed a complete chunk. But again should I do by Database read or File read ?

Update Can I export it using any tool from MySQL database , auto-converted to XML format ?

Answer 1

You can use perl and XML::Simple / XML::Writer perl modules. Perl is also capable of reading data from both database and text files.