Migrating from MySQL plain database values to fully AES encrypted values (PHP APi)

Question

Short:

Best way to migrate from plain-text database values to fully/entirely encrypted values due to GDPR regulations?

Explainer:

I just received a "heads-up" that due to the new "Privacy by design" concept legally required by the GDPR (The EU General Data Protection Regulation) all 'sensitive' information is required to be encrypted. This includes names, addresses, social media profiles, ..

We are currently running a full production database for 2000+ users, only accessible via our PHP REST API that is used as entry point for all web and mobile application (no third-party developers yet). This database contains email addresses, addresses, social media handles, names, ip-addresses .. that are currently stored in plain-text (except for security-based sensitive information like the passwords, tokens,.. or any other value used for authentication/identification).

I'm not a big fan of this (development wise) because of the mayor impact on performance, search algorithms, the whole PHP API etc (feels like an end of the whole optimized and "perfectly setup database"), but since it's required and since it's an extra layer of protection, I'm all in for it.

Now, my main concern is.. everything is actively running and we can't just say "okay, shutting down the server, encrypt everything, deploy the new API version and turn it back on again.". Also, this migration can't be done "step by step along the way" I think, everything has to be done all at once. The database values can't be encrypted if the API isn't ready to handle decryption within all queries, and vice-versa, the database can't be plain text if the API is expecting everything to be encrypted.

(I'm glad I went for the whole API type of deal to be the only point of entry/read for the database, no custom scripts/connections, so this is a relieve)

We're running a highly overpowered VPS server, so a bit of performance-loss due to redundant checks and whatnot is an option and we're running staging/dev environments as well, so testing is no issue.

I would let MySQL handle the AES with a unique key for every record that is based on a combination of the ID (primary key) and the creation timestamp for example (both values that will never change) (which might not be that smart of an idea in case "for some reason" this value does changes, the data is gone.. so not sure about that), so that my PHP application doesn't need to communicate the key in plain text when running queries, but then again, the 'creation of the key for decryption' will still be visible within the query logs etc, so the "most secure" way would be to let the PHP application encrypt and decrypt all incoming/outgoing data, but this would result in not being able to run 'search queries' for example (unless I send the key within those specific queries..).

My question:

How would one go about this..? Should I let PHP handle the en/decryption or preferable MySQL? Is there a usable way to migrate to encrypted values along the way when the record gets requested? ..?

My idea was to no touch any database values (except for updating every non-integer column to handle 2-3x the amount of characters that are currently specified) which can be done without having an impact on production. Then updating all the API queries, step by step, to check if the selected value is encrypted or not. And when one section is running OK, update the table for that section to encrypt all the values. As a (simple) example

MySQL-way; Every SELECT query:

SELECT IF_AES_ENCRYPTED(first_name, AES_DECRYPT(first_name), first_name) AS first_name FROM contacts WHERE id = 1;

(or) PHP-way: Retrieving of data:

while ($row = $result->fetch_assoc()) {
  $contact->setFirstName((IS_AES_ENCRYPTED($row['first_name']) ? AES_DECRYPT($row['first_name']) : $row['first_name']);
}

At the end of the deployment:

UPDATE contacts SET first_name = AES_ENCRYPT(first_name);

There are definitely ways to accomplish this, but since I'm the only developer here at this point, I'm just not sure what would be the most practical/efficient way to do this or if I'm over- or under thinking this or not. Just looking for other developers out there that have executed a migration/update like this.

Thanks, Bert.

Answer 1

I would do this by adding a new column first_name_enc to store the encrypted version of the string.

Then you can store both encrypted and unencrypted values temporarily while you are transitioning to fully encrypted.

ALTER TABLE contacts ADD COLUMN first_name_enc VARBINARY(...);

When you read the values, decrypt it, but if the value is NULL, then it must be a row that hasn't been converted yet, so fall back to the original unencrypted column.

SELECT COALESCE(AES_DECRYPT(first_name_enc, <key-string>), first_name) ...

After you put this code in place throughout your application, then you can start converting the rows in batches:

UPDATE contacts SET
  first_name_enc = AES_ENCRYPT(first_name, <key-string>), 
  first_name = NULL
WHERE id BETWEEN 1 AND 1000;

After you finish converting everything to encrypted, and your application is no longer inserting to the unencrypted columns, then you can convert the conditional queries to simply read the encrypted columns and decrypt them.

SELECT AES_DECRYPT(first_name_enc, <key-string>) ...

Then finally drop the unencrypted columns, since they now contains only NULLs anyway.

ALTER TABLE contacts DROP COLUMN first_name;

---

I recommend before investing time into this work that you confirm what you're really required to do regarding encryption. I've read articles that claim that the GDPR doesn't actually make encryption mandatory. https://www.i-scoop.eu/gdpr-encryption/

But take internet articles with a grain of salt. Consult a qualified expert. Even paying a professional expert a consultation fee might save you tens of thousands in software development costs!