How to search on a table with 50,000 records that are encrypted

Question

I have SQL Server 2012, and I cant migrate to SQL server 2016.

I am using encryption, in this way with Entity Framework Code First.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Configuration;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;

namespace x.y.Api.Models
{
    [Table("Tbl_Naturalezas")] 
    public class Naturalezas: EncryptDecrypt
    {   
        public Naturalezas()
        {
            _locked = true;
        }

        [Key]
        public int idNaturaleza { get; set; }

        string _naturaleza;

        [StringLength(350)]
        public string naturaleza
        {
            get { return locked ? Decrypt(_naturaleza, ConfigurationManager.AppSettings["appKeyPassword"]) : naturaleza; }
            set { _naturaleza = IsEncrypted(value) ? value : Encrypt(value, ConfigurationManager.AppSettings["appKeyPassword"]) ; }
        }

        public virtual ICollection<Contactos> Contactos { get; set; }     

    }
}

Which Inherits from this class:

public class EncryptDecrypt
    {
        public bool _locked;
        public const string EncryptedStringPrefix = "X";

        private const int Keysize = 256;
        private const int DerivationIterations = 1000;

        public string Encrypt(string atributoClase, string passPhrase)
        {
            string plainText = atributoClase.ToUpper();
            if (plainText != null)
            {
                var saltStringBytes = Generate256BitsOfRandomEntropy();
                var ivStringBytes = Generate256BitsOfRandomEntropy();
                var plainTextBytes = Encoding.UTF8.GetBytes(plainText);
                using (var password = new Rfc2898DeriveBytes(passPhrase, saltStringBytes, DerivationIterations))
                {
                    var keyBytes = password.GetBytes(Keysize / 8);
                    using (var symmetricKey = new RijndaelManaged())
                    {
                        symmetricKey.BlockSize = 256;
                        symmetricKey.Mode = CipherMode.CBC;
                        symmetricKey.Padding = PaddingMode.PKCS7;
                        using (var encryptor = symmetricKey.CreateEncryptor(keyBytes, ivStringBytes))
                        {
                            using (var memoryStream = new MemoryStream())
                            {
                                using (var cryptoStream = new CryptoStream(memoryStream, encryptor, CryptoStreamMode.Write))
                                {
                                    cryptoStream.Write(plainTextBytes, 0, plainTextBytes.Length);
                                    cryptoStream.FlushFinalBlock();
                                    // Create the final bytes as a concatenation of the random salt bytes, the random iv bytes and the cipher bytes.
                                    var cipherTextBytes = saltStringBytes;
                                    cipherTextBytes = cipherTextBytes.Concat(ivStringBytes).ToArray();
                                    cipherTextBytes = cipherTextBytes.Concat(memoryStream.ToArray()).ToArray();
                                    memoryStream.Close();
                                    cryptoStream.Close();
                                    return Convert.ToBase64String(cipherTextBytes);
                                }
                            }
                        }
                    }
                }

            }
            else
            {
                return plainText; 
            }
        }

        public string Decrypt(string atributoClase, string passPhrase)
        {
            string cipherText = atributoClase.ToUpper();
            if (cipherText != null)
            {
                // Get the complete stream of bytes that represent:
                // [32 bytes of Salt] + [32 bytes of IV] + [n bytes of CipherText]
                var cipherTextBytesWithSaltAndIv = Convert.FromBase64String(cipherText);
                // Get the saltbytes by extracting the first 32 bytes from the supplied cipherText bytes.
                var saltStringBytes = cipherTextBytesWithSaltAndIv.Take(Keysize / 8).ToArray();
                // Get the IV bytes by extracting the next 32 bytes from the supplied cipherText bytes.
                var ivStringBytes = cipherTextBytesWithSaltAndIv.Skip(Keysize / 8).Take(Keysize / 8).ToArray();
                // Get the actual cipher text bytes by removing the first 64 bytes from the cipherText string.
                var cipherTextBytes = cipherTextBytesWithSaltAndIv.Skip((Keysize / 8)  2).Take(cipherTextBytesWithSaltAndIv.Length - ((Keysize / 8)  2)).ToArray();

                using (var password = new Rfc2898DeriveBytes(passPhrase, saltStringBytes, DerivationIterations))
                {
                    var keyBytes = password.GetBytes(Keysize / 8);
                    using (var symmetricKey = new RijndaelManaged())
                    {
                        symmetricKey.BlockSize = 256;
                        symmetricKey.Mode = CipherMode.CBC;
                        symmetricKey.Padding = PaddingMode.PKCS7;
                        using (var decryptor = symmetricKey.CreateDecryptor(keyBytes, ivStringBytes))
                        {
                            using (var memoryStream = new MemoryStream(cipherTextBytes))
                            {
                                using (var cryptoStream = new CryptoStream(memoryStream, decryptor, CryptoStreamMode.Read))
                                {
                                    var plainTextBytes = new byte[cipherTextBytes.Length];
                                    var decryptedByteCount = cryptoStream.Read(plainTextBytes, 0, plainTextBytes.Length);
                                    memoryStream.Close();
                                    cryptoStream.Close();
                                    return Encoding.UTF8.GetString(plainTextBytes, 0, decryptedByteCount);
                                }
                            }
                        }
                    }
                }
            }
            else
            {
                return cipherText;
            }        
        }

        private static byte[] Generate256BitsOfRandomEntropy()
        {
            var randomBytes = new byte[32]; // 32 Bytes will give us 256 bits.
            using (var rngCsp = new RNGCryptoServiceProvider())
            {
                // Fill the array with cryptographically secure random bytes.
                rngCsp.GetBytes(randomBytes);
            }
            return randomBytes;
        }

        public void Lock()
        {
            _locked = true;
        }

        public void Unlock()
        {
            _locked = false;
        }

        public bool IsEncrypted(string atributosClases)
        {

            if (atributosClases != null)
            {
                if(atributosClases.Length > 50)
                {
                    return true;
                }
                else
                {
                    return false;
                }

            }
            else
            {
                return true;
            }

        }


    }

In the POST api controller, I do this:

// POST: api/Naturalezas
        [ResponseType(typeof(Naturalezas))]
        public IHttpActionResult PostNaturaleza(Naturalezas naturaleza)
        {
            if (!ModelState.IsValid)
            {
                return BadRequest(ModelState);
            }
            naturaleza.Unlock();
            db.Naturalezas.Add(naturaleza);
            db.SaveChanges();
            naturaleza.Lock();
            return CreatedAtRoute("DefaultApi", new { id = naturaleza.idNaturaleza }, naturaleza);
        }

I based this encryption on this blog post:

https://www.tabsoverspaces.com/233147-custom-encryption-of-field-with-entity-framework/?utm_source=blog.cincura.net

Now, this is for one table only, but in another table we have 20 fields and all the fields must be encrypted, however we need to be able to search on those 20 fields using LIKE, =, etc.

What is the best solution (guide me to a code solution), to be able to:

1. Encrypt all fields on the database without using SQL 2016 Always Encrypted.
2. DO Searches.
3. Keep performance.

Answer 1

There are a lot of trade offs you will have to make when encrypting your data at rest with a middle tier solution, as you are currently doing it. While Always Encrypted certainly makes things quite a bit easier and eliminates the custom encryption code in your app, there are still similar limitations to custom encryption, such as not being able do wildcard LIKE filtering because they function similarm which is the encryption/decryption is not happening at the DB level.

A couple of recommendations:

Basic filtering still works When using the same encryption key & salt, you can still perform normal WHERE x = 'y' type of filtering.

Shift Searching & Filtering to Middle Tier

Again, a trade off & can have performance implications, but once the data is decrypted, you can perform more complex filtering with plain old LINQ

Do you really need to encrypt that column?

Is your data not classified as PHI, PII or something similar? Consider not encrypting it and you can perform normal SQL WHERE & LIKE filtering

Answer 2

Well these questions may results on various correct answers. I will just talk about what would I've done for each problem

1. Encrypt all fields on the database.

I think this question can be seperated on 2 parts :

• Part A : Updating your current data to an encrypted format.

This part is the easiest (but not the shortest) and could be solve by a simple oneshot project to read every row and update it to an encrypted format. This part can be optional since your project differentiate encrypted data from clear data.

• Part B : Encrypt your new data

Two choices here, depending on the time you want to give to solve this first question. The best would be to change the way your data is saved from your EF project. The worst would be to re-run the project Part A at given times.

2. DO Searches.

The simpliest and safest here is to load all of your data and request it using Linq.

3. Keep performance.

Their is many approach to this and it depends if you're from a web project or a software project. I will just talk of solution that may concern both.

But beware! Each solution below will add a lot of security problems if you don't take extra care of this point. And implementing many would ask a lot more of tinkering.

• Caching

One of the best generic solution is to have some $cache$. It could be directly caching your database using SqlChangeMonitor ( example ) or with some time-lag from your EF project using Entity Framework Extended .

Maybe you can use both with SqlChangeMonitor to update your EF cache.

• Hinting

Example : Your columnA, B and C has data evenly distributed for each of its possible values. Using these columns with an index and in every Where should give you a quick response.

How to implement : Make a stored procedure with a Select inside or a function returning a table and query only from your result. Your EF project would need to be re-designed in order to query from your function / result set of your stored procedure.

• Use temporary data

At login/entry your user can save all the data in a temporary indexed table with unencrypted data and makes his/her query on this temporary table. It will be easy if you developed the solution 1.A. You can use Keep Fixed Plan or Keep Plan options in order to perform better. Again it will ask you to re-design your EF project (but this one should be simplier).

Warning: Don't use global temporary table. It will break the point to have encrypted data.