Django on Kubernetes Deployment: Best practices for DB Migrations

Question

My Django deployment has x number of pods (3 currently)running a Django backend REST API server. We're still in the development/staging phase. I wanted to ask for advice regarding DB migration. Right now the pods simply start by launching the webserver, assuming the database is migrated and ready. This assumption can be wrong of course.

Can I simply put python manage.py migrate before running the server? What happens if 2 or 3 pods are started at the same time and all run migrations at the same time? would there be any potential damage or problem from that? Is there a best practice pattern to follow here to ensure that all pods start the server with a healthy migrated database?

I was thinking about this:

During initial deployment, define a Kube.netes Job object that'll run once, after the database pod is ready. It will be using the same Django container I have, and will simply run python manage.py migrate . the script that deploys will kubectl wait for that job pod to finish, and then apply the yaml that creates the full Django deployment. This will ensure all django pods "wake up" with a database that's fully migrated.

In subsequent updates, I will run the same job again before re-applying the Django deployment pod upgrades.

Now there is a question of chicken and egg and maintaining 100% uptime during migration, but this is a question for another post: How do you apply data migrations that BREAK existing container Version X when the code to work with the new migrations is updated in container Version X+1. Do you take the entire service offline for the duration of the update? is there a pattern to keep service up and running?

Answer 1

Well you are right about the part that multiple migrate commands will run against your database by multiple pods getting started.

But this will not cause any problems. When you are going to make actual changes to your database, if the changes are already applied, your changes will be ignored. So, say 3 pods start at the same time and run the migrate command. Only One of those commands will end up applying changes to the database. Migrations normally need to lock the database for different actions (this is highly related to your DBMS). The lock will happen by one of the migrate commands (one of the pods) and other commands should wait until the work of the first one is over. After the job is done by the first one, others' commands will be ignored automatically. So each migration will happen once.

You can however, change your deployment strategy and ask kubernetes to first, spin up only 1 pod and when the first pod's health check succeeds, others will spin up too. In this case, you can be sure that the lock time for the migration, will happen only once and others will just check that migrations are already applied and ignore them automatically.

Answer 2

You may use Kubernetes init containers, which are specialized containers that run before app containers in a Pod. The init containers stop after successfully executing the commands you want, so they won't occupy unnecessary resources. Here is the official link: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

Answer 3

The best thing you can do is to use kind:Jobs, an run this before the rollout. Please check: Django migrations by Kubernetes Job and persistent Volume Claim

Answer 4

As I understand In K8s you can't control the order of the pods. Let's say your DB host is a docker container (Dev environment) and not an RDS/ static host, migrations command could not be executed before the DB is up and ready.

I try to to use the ready function in app.py to execute run_migrations.py that execute shell script of run migrations but run migrations trigger app.py and it's infinity loop.

In my organization We use circleCI which build the image first and then push it to AWS ECR with tags before deploy it to EKS. At the build stage I can't assume the DB pod is ready so run migrations will cause an error.

When the DB for Django is also a K8s pod, the best way to execute it will be a job.