从包中两个相关表中下载行的最快方法是什么？

Question

I have some problems with two massive related tables. First one has about 100 columns, second one about 300. The foreign key is on 5 columns. 100 million rows is nothing special in these tables.

My task is to download all the rows, process them and then upload. I need to download these tables in packs (10000 rows from parent table + all related to them rows from second table).

What would be the fastest way to do it?

Simplest solution would be downloading 10000 rows from parent table and then iterating through them to download related rows. Simple but I don't think it will be fast.

Other solution could be download with joining those two tables. Problem is that then I have to separate row in two parts, eliminate duplicates, etc. I also don't really know how fast this download would be.

So, my question is the same as in the title. What is the fastest way to download massive ammount of data from related tables in packs?

Answer 1

I think the best solution here is to firstly download all the rows you require. SO all 1mil rows and put those into a List where Type is the actual type of the Table. This is easily done using a framework like NHibernate where you can map database structure to classes.

Once you have that then you can proceed by something like this: You have a number of batches lets say 10,000 per batch.

    int totalCount = LIST.Count;
                int batchSize = 10000;
                int numberOfBatches = (int)Math.Ceiling((decimal)totalCount / batchSize);

     for (int i = 0; i < numberOfBatches; i++)

                    {
    var currentBatch = LIST.Skip(i * batchSize).Take(batchSize);

CONTENT HERE.

}

So basically you will be updating x number of rows at a time in the database. I highly suggest you use NHibernate as opposed to SQLReader/Writers as they are much more efficient and tasks like insert/update/delete become trivial.

EDIT: Alternatively to NHibernate Update you can use BulkUpdate Have a look at Bulk Update in C# .

Answer 2

The fastest way would be to use an ETL tool like SSIS to process the data on the server without transferring it to other machines.

SSIS allows batching, per-row processing of data streams with many built-in operations or even C# scripts, execution monitoring, handling of dirty data etc.

In ETL scenarios IO is the big killer, so transferring the data to other machines should be avoided. Connection latencies are another killer, so retrieving child record from a client machine is also going to kill performance.

A proper SQL statement like a join between two tables will perform orders of magnitude better than pulling the data to some other machine then pushing it back. Moreover, the database can optimize large selects and updates because it can select the proper query and update strategies and uses only the data that is actually used in selects or updates.

Finally, ORMs like NHibernate, EF or Linq to SQL should be avoided at all costs in ETL scenarios with even modest data sizes. The CPU and memory overhead of mapping thousands (much less millions) of objects is significant without providing any benefit.