分块XML并将其加载到关系表中

Question

I work for a credit union (roughly 60K accounts). The statement process is from the '70s and it tightly coupled the data to the layout. In short, you run a job and it produces a text file that contains the statement for each account. I've modified the mainframe config and now instead of getting text out, I get XML like so:

<statements>
    <statement account='1'>
       ...statement info like checking/savings/certificate/visa/loan/heloc shares
    </statement>
    <statement account='N'>
       ...statement info like checking/savings/certificate/visa/loan/heloc shares
    </statement>
</statements>

I wrote java code to pull data from relational table(s) and build PDFs on the fly with iText. Some of the data that shows up on the statement is calculated from data in the XML. For instance, the XML contains all the transactions on the share. On the statement, we want to show the number of credits and the number of debits. Once loaded into a DB, I can use a view to calculate these values on the fly and provide the data to my java app.

this XML file is ~900MB and only going to grow as we add more members.

I want to process the xml one "statement" at a time. http://mrico.eu/entry/parsing_chunks_of_xml_documents

Can JAXB parse large XML files in chunks )

Once I have an individual statement, I want to load it's shares (checking, savings, visa, etc) into corresponding DB tables.

Seems like the simplest way to accomplish this is to bind the statement to a POJO and then for each complex element (share or transaction or loan) in the POJO, do an insert.

what combination of parser / binder / persistence tools would you guys recommend?

personally, I'd favor raw JDBC inserts, so the question of parser and binder is more important.

Note: I could probably create a schema for the XML, but it might be fragile due to the way the mainframe builds the XML file. Anyone using Fiserv's Spectrum software feels my pain.

Answer 1

看一下StAX ，它是XML的流API。