Decrypting data and exporting to excel

Question

I've got a SQL server DB with thousands of rows of encrypted data, not all columns are encrypted though. I've also got an excel exporter in place that, when activated, goes through each row, checks each column of that row to see if it is one of the encrypted cols, decrypts it if so and then moves on.

The problem is that this involves a nested foreach and a checking system that's is really slow and only going to get worse as the DB gets larger.

This is a sample of the section of code that is really slow, in the below code 'encryptedColumns' is a List<> of strings, the columns names of those that are encrypted.

This is obviously inefficient but can anyone suggest a quick fix to speed it up?

foreach (DataRow dr in dtData.Rows)
{
    foreach (DataColumn c in dtData.Columns)
    {
        if (encryptedColumns.Contains(c.ColumnName.ToLower()))
        {
            if (!String.IsNullOrEmpty(dRow[c].ToString()))
            {
                string decryptedTitle = Encryption.DecryptStringAES( dRow[c].ToString(), 'AESKEY goes here');
                dRow[c] = decryptedTitle;
            }
        }
    }
}

Bit more detail. The above is for the dynamic creation of an excel document, I iterate through all rows and when I reach a column that requires decryption I decrypt it and so on.

An example row might be, where you can see the obviously encrypted content among the plain text content.

Joe | Bloggs | "£$D£"$ | Blue | "£$%^FG£$DF"£$

Answer 1

Instead of iterating through each row and checking each column (which is of O(nxm) best-case time complexity, as you always check n rows and m columns), I would iterate via columns, find the ones that need decryption and decrypt those values.

Average time complexity of the column-first approach is O(Const xn) ~ O(n) , where Const is number of columns that need decryption and n is number of rows. That is, you only iterate on the columns that need decryption and do not check any other rows/columns.

I'd also use search and pagination - display part of the table and allow user to go back and forth. No one can understand a table with millions of rows thrown at a user, but if you present, say 100 rows at a time, allow pagination and search on unencrypted data that will solve both performance and usability problems.

Yet another thing you can try is to run decryption concurrently, providing you don't modify data, and hence no need for locking, you can now run several threads decrypting different parts of the dataset for you.