Error in Kubernetes cronjob to dump and restore postgres database for Django application

Question

I need to create a testing environment for a specific team to use our Django application for testing against the database. This database must be somewhat in sync with out production database. Considering we have two clusters production and staging , my approach is the following:

• Create a db_test in the staging Postgres server
• db_test will be periodically synced with db_prod . To do so:
• Create a cronjob in the staging cluster that connects to the production database, do a pg_dump and then do a pg_restore to the db_test (using localhost because it's connected through a pgbouncer).
• I run the migrations to make sure the tables are up to date
• Next: I need to run a management command that will erase some customer data from this copy (to be GDPR compliant).

Desired behavior:

• pg_dump and pg_restore are successful and the new database is cleared of customer information

Actual behaviour:

• pg_dump and pg_restore are successful. I can psql into the newly created database and everything looks alright.
• migrate command fails with the traceback below
• clean_db fails because it can't find some tables (which they exist as I inspected with psql .

This is the simple shell script I run in the cronjob:

#!/bin/bash
# Dump the database locally
pg_dump --host=mydb.postgres.database.azure.com \
        --username=myuser@production \
        --no-owner \
        --verbose \
        -Ft db_prod > $HOME/db_prod-$(date +%F).tar &&

sleep 30 &&

# Restore the database
pg_restore --no-owner \
    --no-acl \
    --host='localhost' \
    --user=myuser@staging \
    --port=5432 \
    --clean \
    --dbname=db_test \
    $HOME/db_prod-$(date +%F).tar \
    --verbose &&

python manage.py migrate &&

python manage.py clean_db

• This is the error of the clean_db without the migrate before

relation "uploader_somemodel" does not exist
2
LINE 1: SELECT COUNT(*) AS "__count" FROM "uploader_somemodel...

But this is the log of the table being created

pg_restore: processing data for table "public.uploader_somemodel"```

• And this is the error log of the migrate command

Running all migrations...
Operations to perform:
  Apply all migrations: admin, auth, axes, contenttypes, django_mfa, myapp, reversion, sessions, uploader, userapi
Running migrations:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 67, in ensure_schema
    editor.create_model(self.Migration)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
    self.execute(sql, params or None)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
    cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: no schema has been selected to create in
LINE 1: CREATE TABLE "django_migrations" ("id" serial NOT NULL PRIMA...
                     ^

Has anyone ever encountered a similar error and know what this is about? Or even better, do you have a suggestion of how I could perform this diferently?

Answer 1

Did you change the NAME of the db in your env variable ? To get this result in your django settings.py:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': db_test,
        'USER': POSTGRES_USER,
        'PASSWORD': POSTGRES_PASSWORD,
        'HOST': POSTGRES_HOST,
        'PORT': 5432,
    }
}