How can I migrate AWS RDS database through eb cli (django postgres)?

Question

I have a django project deployed with elastic beanstalk CLI. I added a postgres database (AWS RDS), the connection is runnning.

My database is still empty. How can I run migration commands or "python manage.py createsuperuser" inside of eb shell?

When I open eb ssh <env> and try ls , nothing is shown. Stepping one directory back with cd.. and again ls shows me ec2-user , healthd and webapp . I have not the permissions to enter these folders. Isnt it possible, to run python commands inside of eb ssh?

I did also try container commands, .ebextensions/db-migrate.config:

container_commands:
  01_makemigrations:
    command: "python manage.py makemigrations"
    leader_only: true
  02_migrate:
    command: "python manage.py migrate --first main initial && python manage.py migrate"
    leader_only: true
option_settings:
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: <app>.settings

After commiting and deploying again with eb deploy , I get this error:

eb deploy Creating application version archive "app-xxxx". Uploading: [##################################################] 100% Done...

2021-02-22 12:57:12 INFO Environment update is starting.

2021-02-22 12:58:05 INFO Deploying new version to instance(s).

2021-02-22 12:58:21 INFO Instance deployment successfully generated a 'Procfile'.

2021-02-22 12:58:23 ERROR Instance deployment failed. For details, see 'eb-engine.log'.

2021-02-22 12:58:24 ERROR [Instance: i-xxxx] Command failed on instance. Return code: 1 Output: Engine execution has encountered an error..

2021-02-22 12:58:24 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].

2021-02-22 12:58:24 ERROR Unsuccessful command execution on instance id(s) 'i-xxxx'. Aborting the operation.

2021-02-22 12:58:24 ERROR Failed to deploy application.

ERROR: ServiceError - Failed to deploy application.

Why are the commands failing? Any advice is appreciated. Btw, I have no problems with deploying with the local sql db.

UPDATE:

eb-engine.log refers me to cfn-init.log. This is the output:

2021-02-22 16:49:43,624 [ERROR] Command 01_makemigrations (python manage.py makemigrations) failed

2021-02-22 16:49:43,624 [ERROR] Error encountered during build of postbuild_0_django_test: Command 01_makemigrations failed

Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog)

File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build changes['commands'] = CommandTool().apply(self._config.commands)

File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply raise ToolError(u"Command %s failed" % name) ToolError: Command 01_makemigrations failed

In the Path I see python 2.7 (?) but actually Python 3.7 is running on 64bit Amazon Linux 2. I also tried executing the command with "python3".

Answer 1

The issue is that django is not loaded at that point.

Solution:

container_commands:
  01_migrate:
    command: |
      source $PYTHONPATH/activate
      pipenv run python ./manage.py migrate

Logs referring you to eb-engine.log and to cfn-init.log , but I found the final hint in cfn-init-cmd.log :

2021-02-23 11:08:34,019 P5022 [INFO] ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?

I did also try it for "createsuperuser". No errors at all, but also no new user created. There is no active command line to enter username/password. I try to update my answer if I found a solution for createsuperuser.

Answer 2

I can propose an alternate approach:

Rather than running Django management commands from within the eb shell, you could adjust the DATABASE settings of your local Django project to the remote database, and execute management commands from your local terminal.

This would be useful for running commands ad-hoc, such as createsuperuser , but for routine maintenance commands, like migrate , you'd want to stick with the container_commands approach as part of your deployment process.

Finally, if you're determined to run createsuperuser , specifically, within the eb shell, then you can use container_commands to do so non-interactively, with some additional settings configured. From the Django Docs :

When run interactively, this command will prompt for a password for the new superuser account. When run non-interactively, you can provide a password by setting the DJANGO_SUPERUSER_PASSWORD environment variable. Otherwise, no password will be set, and the superuser account will not be able to log in until a password has been manually set for it.

In non-interactive mode, the USERNAME_FIELD and required fields (listed in REQUIRED_FIELDS) fall back to DJANGO_SUPERUSER_<uppercase_field_name> environment variables, unless they are overridden by a command line argument. For example, to provide an email field, you can use DJANGO_SUPERUSER_EMAIL environment variable.