使用C＃如何从SQL数据库中有效检索和更新数千条记录

Question

Process

1. I am writing a C# application which will need to retrieve 4 million records(ID) from a SQL table in database A.
2. I then need to use each ID to select a row of record each from another SQL table in database B.
3. Once I have this row I then need to update another SQL table in database C

Questions

1. What's the most efficient way to retrieve and store the data in Step 1? a. Should I load this in a list string? b. Do you recommend doing batches initially?

2. What the most efficient way to achieve steps 2 and 3

Answer 1

To retrieve the 4M records you're going to want to use a SqlDataReader - it only loads one row of data into memory at a time.

var cn = new SqlConnection("some connection string");
var cmd = new SqlCommand("SELECT ID FROM SomeTable", cn);
var reader = cmd.ExecuteReader();

while (reader.Read())
{
    var id = reader.GetInt32(0);

    // an so on
}

reader.Close();
reader.Dispose();
cn.Close();

Now, to handle two and three, I would leverage a DataTable for the row you need to retrieve and then a SqlCommand on the third database. This means that inside the reader.Read() you can get the one row you need by filling a DataTable with a SqlDataAdapter and the issuing an ExecNonQuery against a SqlCommand for the UPDATE statement.

Another way of writing the above, and it's a bit safer, is to use the using statement:

using (SqlDataReader reader = cmd.ExecuteReader())
{
    while (reader.Read())
    {
        var id = reader.GetInt32(0);

        // an so on
    }
}

that will eliminate the need for:

reader.Close();
reader.Dispose();

and so you could also issue that for the SqlConnection if you wanted.

Answer 2

SQLBulkCopy class might help.

http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx