SqlBulkCopy WriteToServer 使用 IDataReader 而不是 DataTable 和以编程方式调整的字段值

Question

We have a working code in C# that utilizes SqlBulkCopy to insert records into a table from a stored procedure source. At a high-level:

1. Reads data from a stored procedure that puts the records into a DataTable. Essentially calls the SP and does an AdpAdapter to put the records into the DataTable. Let's call this srcDataTable.
2. Dynamically maps the column names between source and destination through configuration, a table that's similar to the following:

TargetTableName	ColumnFromSource	ColumnInDestination	DefaultValue	Formatting
TableA	StudentFirstName	FirstName	NULL	NULL
TableA	StudentLastName	LastName	NULL	NULL
TableA	Birthday	Birthdate	1/1/1900	dd/MM/yyyy

3. Based on the mapping from #2, set up new rows from srcDataTable using .NewRow() of a DataRow to another DataTable that matches the structure of the destination table (where ColumnNameOfDestination is based). Let's call this targetDataTable. As you can see from the table, there may be instances where the value from the source is not specified, or needs to be formatted a certain way. This is the primary reason why we're having to add data rows on the fly to another data table, and the adjustment / defaulting of the values are handled in code.
4. Call SqlBulkCopy to write all the rows in targetDataTable to the actual SQL table.

This approach has been working alright in tandem with stored procedures that utilize FETCH and OFFSET so it only returns an X number of rows at a time to deal with memory constraints. Unfortunately, as we're getting more and more data sources that are north of 50 million rows, and that we're having to share servers, we're needing to find a faster way to do so while keeping memory consumption in check. Researching options, it seems like utilizing an IDataReader for SQLBulkCopy will allow us to limit the memory consumption of the code, and not having to delegate getting X number of records in the stored procedure itself anymore.

In terms of preserving current functionality, it looks like we can utilize SqlBulkCopyMappingOptions to allow us to maintain mapping the fields even if they're named differently. What I can't confirm however is the defaulting or formatting of the values.

Is there a way to extend the DataReader's Read() method so that we can introduce that same logic to revise whatever value will be written to the destination if there's configuration asking us to? So a) check if the current row has a value populated from the source, b) default its value to the destination table if configured, and c) apply formatting rules as it gets written to the destination table.

Answer 1

You appear to be asking "can I make my own class that implements IDataReader and has some altered logic to the Read() method?"

The answer's yes; you can write your own data reader that does whatever it likes in Read(), format the server's hard disk as soon as it's called even.. When you're implementing an interface you aren't "extend[ing] the DataReader's read method", you're providing your own implementation that externally appears to obey a specific contract but the implementation detail is entirely up to you. If you want, upon every read, to pull down a row from db X into a temp array, zip through the array tweaking the values to have some default or other adjustment, before returning true that's fine..

..if you wanted to do the value adjustment in the GetXXX, then that's also fine.. you're writing the reader so you decide. All the bulk copier is going to do is call Read until it returns false and write the data it gets from eg GetValue (if it wasn't immediately clear: read doesn't produce the data that will be written, GetValue does. Read is just an instruction to move to the next set of data that must be written but it doesn't even have to do that. You could implement it as { return DateTime.Now.DayOfWeek == DayOfWeek.Monday; } and GetValue as { return Guid.NewGuid().ToString(); } and your copy operation would spend until 23:59:59.999 filling the database with guids, but only on Monday)

Answer 2

The question is a bit unclear. It looks like the actual question is whether it's possible to transform data before using SqlBulkCopy with a data reader.

There are a lot of ways to do it, and the appropriate one depends on how the rest of the ETL code does. Does it only work with data readers? Or does it load batches of rows that can be modified in memory?

Use IEnumerable<> and ObjectReader

FastMember's ObjectReader class creates an IDataReader wrapper over any IEnumerable<T> collection. This means that both strongly-typed .NET collections and iterator results can be sent to SqlBulkCopy.

IEnumerable<string> lines=File.ReadLines(filePath);

using(var bcp = new SqlBulkCopy(connection)) 
using(var reader = ObjectReader.Create(lines, "FileName")) 
{ 
  bcp.DestinationTableName = "SomeTable"; 
  bcp.WriteToServer(reader); 
}

It's possible to create a transformation pipeline using LINQ queries and iterator methods this way, and feed the result to SqlBulkCopy using ObjectReader . The code is a lot simpler than trying to create a custom IDataReader .

In this example, Dapper can be used to return query results as an IEnumerable<> :

IEnumerable<Order> orders=connection.Query<Order>("select ... where category=@category",
                                                  new {category="Cars"});

var ordersWithDate=orders.Select(ord=>new OrderWithDate {
    ....
    SaleDate=DateTime.Parse(ord.DateString,CultureInfo.GetCultureInfo("en-GB");
});

using var reader = ObjectReader.Create(ordersWithDate, "Id","SaleDate",...));

Custom transforming data readers

It's also possible to create custom data readers by implementing the IDataReader interface. Libraries like ExcelDataReader and CsvHelper provide such wrappers over their results. CsvHelper's CsvDataReader creates an IDataReader wrapper over the parsed CSV results. The downside to this is that IDataReader has a lot of methods to implement. The GetSchemaTable will have to be implemented to provide column and information to later transformation steps and SqlBulkCopy.

IDataReader may be dynamic, but it requires adding a lot of hand-coded type information to work. In CsvDataReader most methods just forward the call to the underlying CsvReader, eg :

public long GetInt64(int i)
{
            return csv.GetField<long>(i);
}

public string GetName(int i)
{
    return csv.Configuration.HasHeaderRecord
        ? csv.HeaderRecord[i]
        : string.Empty;
}

But GetSchemaTable() is 70 lines, with defaults that aren't optimal. Why use sting as the column type when the parser can already parse date and numeric data for example?

One way to get around this is to create a new custom IDataReader using a copy of the previous reader's Schema Table and adding the extra columns. CsvDataReader 's constructor accepts a DataTable schemaTable parameter to handle cases where its own GetSchemaTable isn't good enough. That DataTable could be modified to add extra columns :

    /// <param name="csv">The CSV.</param>
    /// <param name="schemaTable">The DataTable representing the file schema.</param>
    public CsvDataReader(CsvReader csv, DataTable schemaTable = null)
    {
        this.csv = csv;

        csv.Read();

        if (csv.Configuration.HasHeaderRecord)
        {
            csv.ReadHeader();
        }
        else
        {
            skipNextRead = true;
        }

        this.schemaTable = schemaTable ?? GetSchemaTable();
    }

A DerivedColumnReader could be created that does just that in its constructor :

public DerivedColumnReader<TSource,TResult>(string sourceName, string targetname,Fun<TSource,TResult> func,DataTable schemaTable)
{
...
  AddSchemaColumn(schemaTable);
  _schemaTable=schemaTable;
}

void AddSchemaColumn(DataTable dt,string targetName)
{
    var row = dt.NewRow();
    row["AllowDBNull"] = true;
    row["BaseColumnName"] = targetName;
    row["ColumnName"] = targetName;
    row["ColumnMapping"] = MappingType.Element;              
    row["ColumnOrdinal"] = dt.Rows.Count+1;
    row["DataType"] = typeof(TResult);

    //20-30 more properties
    dt.Rows.Add(row);
}

That's a lot of boiler plate that's eliminated with LINQ.

Answer 3

Just providing closure to this. So the main question really is to how we can avoid running into out of memory exceptions when fetching data from SQL without employing FETCH and OFFSET in the stored procedure. The resolution didn't require getting fancy with a custom Reader similar to SqlDataReader, but adding count checking and calling SqlBulkCopy in batches. The code is similar to what's written below:

using (var dataReader = sqlCmd.ExecuteReader(CommandBehavior.SequentialAccess))
                    {
                        int rowCount = 0;

                        while (dataReader.Read())
                        {
                            DataRow dataRow = SourceDataSet.Tables[source.ObjectName].NewRow();
                            for (int i = 0; i < SourceDataSet.Tables[source.ObjectName].Columns.Count; i++)
                            {
                                dataRow[(SourceDataSet.Tables[source.ObjectName].Columns[i])] = dataReader[i];
                            }
                            SourceDataSet.Tables[source.ObjectName].Rows.Add(dataRow);
                            rowCount++;

                            if (rowCount % recordLimitPerBatch == 0)
                            {
                                // Apply our field mapping
                                ApplyFieldMapping();

                                // Write it up
                                WriteRecordsIntoDestinationSQLObject();

                                // Remove from our dataset once we get to this point
                                SourceDataSet.Tables[source.ObjectName].Rows.Clear();
                            }
                        }
}

Where ApplyFieldMapping() makes field-specific changes to the contents of the datatable, and WriteRecordsIntoDestinationSqlObject(). This allowed us to call the stored procedure just once to fetch the data, and let the loop keep memory in check by writing records out and clearing them afterwards when we hit a preset recordLimitPerBatch.