SSIS-插入大量行（亿万行）的最佳方法

Question

So here's the scenario: I have an XML file, which is in size of 500GB, and with data of around 600 million rows (once on a database table). I'm using SSIS for the operation, and since it consumes a REALLY large amount of memory if I am to use an SSIS component (ie: XML Source ), it might cause a timeout (correct me if I'm wrong, but as far as I know, using the components on SSIS loads the content of the XML into memory - with a file that big it will surely cause errors). My approach then is:

• Use a Script Task to parse the XML data using XML Reader (XML Reader by far is the best approach, since it parses the XML on a forward, non-cached approach)
• Insert the data on a DataTable
• Every 500,000 rows on the DataTable , insert the contents to the database using SqlBulkCopy , then clear the contents of the DataTable

My problem is, currently, I tried it to parse another file with the size of 200GB, and it's running on around 13.5M / 1 hour - and I don't know if it's still fine with that run time. It sure solves my problem - but it's not too elegant, I mean, there should be other ways.

I'm looking on other approaches, like:

• Dividing the large XML files into small pieces of CSVs (around 20GB) then use an SSIS Data Flow task
• Use INSERT script every new rows

Can you help me do decide which is best? Or suggest any other solutions.

Every answer will be very much appreciated.

EDIT

I forgot to mention - my approach will be dynamic. I mean, there are many tables that will be populated with large sized XML files. So, using a Script Component as source might be not so useful, since I still need to define the output columns. But still, will give it a try.

EDIT 2015-07-28

The file is from our client, and we can't do anything on what source they want to send to us. XML, that's it. Here is a sample from the XML I am consuming:

<?xml version="1.0" encoding="UTF-8"?>
<MFADISDCP>
  <ROW>
    <INVESTMENT_CODE>DATA</INVESTMENT_CODE>
    <DATE_OF_RECORD>DATA</DATE_OF_RECORD>
    <CAPITAL_GAIN_DISTR_RATE>DATA</CAPITAL_GAIN_DISTR_RATE>
    <INCOME_DISTR_RATE>DATA</INCOME_DISTR_RATE>
    <DISTR_PAYMENT_DATE>DATA</DISTR_PAYMENT_DATE>
    <CURRENCY>DATA</CURRENCY>
    <CONFIRM>DATA</CONFIRM>
    <EXPECTED_DISTRIBUTION_AMOUNT>DATA</EXPECTED_DISTRIBUTION_AMOUNT>
    <KEYING_STATUS>DATA</KEYING_STATUS>
    <DAF_RATE>DATA</DAF_RATE>
    <INCOME_START_DATE>DATA</INCOME_START_DATE>
    <ALLOCABLE_END_DATE>DATA</ALLOCABLE_END_DATE>
    <TRADE_DATE>DATA</TRADE_DATE>
    <OVR_CAPITAL_GAIN_DISTR_OPTION>DATA</OVR_CAPITAL_GAIN_DISTR_OPTION>
    <OVR_INCOME_DISTR_OPTION>DATA</OVR_INCOME_DISTR_OPTION>
    <BACKDATED_DISTRIBUTION>DATA</BACKDATED_DISTRIBUTION>
    <DATE_MODIFIED>DATA</DATE_MODIFIED>
  </ROW>
<!--AROUND 49M+ OF THIS ROWS-->
</MFADISDCP>

Answer 1

If I were to do this then I would break it down into the following tasks:

1. Convert XML file into a (tab or comma) delimited file. If your server has fast disks (SSD) then this should be very quick. Be careful of strings in your data that may contain special characters that may break the delimiter format. Don't use the DataTable object as it is slow. You could stream this so that you don't need to have the whole file in memory at one go (unless your server has several hundred gigs of memory)
2. Truncate the stage table in your database that you will use to load the data into.
3. Use SQL Server's bcp.exe to push the delimited file into a stage table on your database. This is probably the fastest way to get a large amount of data into a database. A problem with this is that if it fails then it is very hard to find which row of data caused the failure.
4. Delete the delimited file as you don't need them lying around taking up lots of space.
5. Create a SQL stored procedure to move the data from the stage table to wherever you will be using it.

You could use SSIS script tasks for this or you could write your own stand alone services.

Note, this is all theoretical, there may be better ways of doing it but this may be a good starting point and to find out where your bottlenecks are.