Multi System Database structure based copying/updating best practice

Question

so after searching and not finding similar cases I want to open a new question. So here is the case: We are working with a large database with a very complicated data structure. Also we are working on multiple systems to ensure stability (development, testing, quality and productive) and its always a struggle so move data between those systems. As I said the data structure is very large and there is also a lot of logic inside the database. Customers are able to add new data parts as configuration and there is also a static income of data which are used for statistics and monitoring. So let me explain the problem with a small example:

Lets take this Database as an example. We have some families making some contest with each other. And they will create some statistics about the points they make.

• The Purple Tables are fixed configurations. They are created once and they can only be changed via an Operator. Those changes will be done and tested in the development system first.
• The Yellow Tables are changing configurations. Each Family is able to create or delete multiple Contests and assign their kids.
• The Red Table is just plain data. Each time a kid makes points, a new row is added with the amount and current time and the relation to the kid and contest. This table will be the base for the later statistics.

This Database is developed on two systems a productive one which is used by the families and a develop system which is used by the programmers/operators.

While developing the programmers will add test data like kids families contests and points. And while using the families will create new contests and assign new kids and will fill up the point table.

• It's necessary to copy new/tested/fixed families from the development to the productive system.
• Its also necessary to copy Contests, Contest-Kid-Assignments and Points from the productive to the development system to find new errors.
• Also it must be possible to change the table structure on the development system and transmit this change to the productive system. (This shouldn't be the main topic here sometimes it can be such a large changes that there just is no easy way, so lets keep this point simple but keep it in mind.)

I want to copy parts of the tables to another system but be able to ignore some tables (for example: Points) and I want to make sure to not copy kids without their parent family so there is no "parentless" object in the database.

Question: What would be a good and save way to do this?

I don't need a solution for a specific database type or some scripts. I'm looking for tools, libraries or good practice. (But just as a note we're using mssql.)

We are currently making a tool for this problem (not going well: unstable, overly complicated, slow and possible reinventing the wheel).

Also a lot of devs I know just copy the whole database (making a backup and running it into another server) But this is also making problems: users are being copied and their guid change so they loose permissions etc. I don't think this is a good solution. Also the database is down for quite a long time and its never a smooth process.

Making it manually is sometimes the easiest way but considering the size of our data structure its not just a huge piece of work there is also a large possibility for mistakes.

So I'm hoping someone knows a tool or something similar to help me out.

Answer 1

Welcome to the pains of development for a Stateful entity like a database. :) RedGate makes a tool called SQL Source Control that is good for moving changed data and Schema into Production, and it can interface with source control solutions such as GIT. It's a bit pricey, but it's the best I've found. One option for keeping dev up to date with prod data and dev changes is one I concocted at my last place of employment, which was... not 100% perfect, but better than nothing, and free. It was developed in Powershell, and it went something like this:

1. Create Pre-restore, Pre-dacpac and Post-dacpac SQL scripts to store data and permission diffs between dev and prod
2. Use SQLPackage.EXE to make DacPac of Dev(Dacpac is basically an xml schema of db, no data)
3. Execute Pre-restore Proc (Often copying out test data that needs to be persisted)
4. Restore Prod over Dev
5. Execute Pre-dacpac script (any DDL That could cause data loss may need to go here)
6. Use SQLPackage.EXE to apply DacPac made in step 2 to Newly restored database
7. Execute Post-Dacpac Script (Permissions, restoration of data copied in step 3)

Again, like I said, it worked and automated the restoration of prod data into our dev environment while keeping our dev changes intact, but it required a good bit of upkeep and maintenance. Also, keep in mind, once your DB reaches a certain size, doing a nightly restore is no longer a viable option due to the time it takes to restore.