Unable to read env variables when running a python code via shell script

Question

I have a python script that I am hosting in an EC2 instance (using CI, CodeDeploy, CodePipeline). In the code, I am taking the path of the DB as env variable as follows:

def db_connection():
    DB_ADAPTER = os.environ.get('DB_ADAPTER')
    DB_USER = os.environ.get('DB_USER')
    DB_PASSWORD = os.environ.get('DB_PASSWORD')
    DB_HOST = os.environ.get('DB_HOST')
    DB_NAME = os.environ.get('DB_NAME')
    engine_url = DB_ADAPTER + '://' + DB_USER + ':' + \
        DB_PASSWORD + '@' + DB_HOST + '/' + DB_NAME
    eng = db.create_engine(engine_url)
    conn = eng.connect()
    print('Connected to the DB')
    return eng, conn

I launched the instance, ran it and did nano.profile . In the .profile , I added the following lines manually:

export DB_ADAPTER=postgresql+psycopg2
export DB_USER=dummy_user
export DB_PASSWORD=dummy_pwd
export DB_HOST=ec2-xx-xxx-xxx-xxx.eu-central-1.compute.amazonaws.com
export DB_NAME=dummy_db

When I push the code to the GitLab repo, the CI runs and the code is dumped as a zipped file in the S3 bucket and then the CodeDeploy and CodePipeline starts.

It is during this stage I get the following error:

The start_script.sh is:

#!/usr/bin/env bash
cd /home/ubuntu/anomaly-detection/
python3.7 ad_fbprophet.py
exit

and appspec.yml :

version: 0.0
os: linux
files:
  - source: /
    destination: /home/ubuntu/anomaly-detection
permissions:
- object: /
  pattern: "*"
  owner: root
  group: root
hooks:
  BeforeInstall:
    - location: /scripts/before_install.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: /scripts/start_script.sh
      runas: ubuntu

But, when I login to the EC2 instance from my laptop and run the python script, it runs perfectly without any error and gives the output.

Any help is appreciated.

Answer 1

I might be wrong here, but as far as I know, by editing.profile file you specify what env variables to export only when you actually login and run the shell with the user for whom you edited the.profile . It appears that you start your job as a 'root' for whom the.profile might look entirely different. I guess you'd have to 'source' the particular profile you're interested in on startup to see the env variables.

$ source /home/your_user/.profile

Alternatively, try adding the env variables to /etc/profile instead of your home profile.

Answer 2

You must install python-dotenv

You can do that with this command:

pip install python-dotenv

Then in your code you must add:

from dotenv import load_dotenv

load_dotenv()