Django migration 11 million rows, need to break it down

Question

I have a table which I am working on and it contains 11 million rows there abouts... I need to run a migration on this table but since Django trys to store it all in cache I run out of ram or disk space which ever comes first and it comes to abrupt halt.

I'm curious to know if anyone has faced this issue and has come up with a solution to essentially "paginate" migrations maybe into blocks of 10-20k rows at a time?

Just to give a bit of background I am using Django 1.10 and Postgres 9.4 and I want to keep this automated still if possible (which I still think it can be)

Thanks Sam

Answer 1

The issue comes from a Postgresql which rewrites each row on adding a new column (field).

What you would need to do is to write your own data migration in the following way:

1. Add a new column with null=True . In this case data will not be rewritten and migration will finish pretty fast.
2. Migrate it
3. Add a default value
4. Migrate it again.

That is basically a simple pattern on how to deal with adding a new row in a huge postgres database.