How to set environmental variables properly Gitlab CI/CD and Docker

Question

I am new to Docker and CI/CD with Gitlab CI/CD . I have .env file in the root directory of my Django project which contains my environmental variable eg SECRET_KEY=198191891 . The .env file is included in .gitignore . I have set up these variables in Gitlab settings for CI/CD. However, the environment variables set in Gitlab CI/CD settings seem to be unavailable

Also, how should the Gitlab CI/CD automation process create a user and DB to connect to and run the tests? When creating the DB and user for the DB on my local machine, I logged into the container docker exec -it <postgres_container_name> /bin/sh and created Postgres user and DB.

Here are my relevant files.

docker-compose.yml

version: "3"
services:
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data/
  web:
    build: .
    command: /usr/local/bin/gunicorn writer.wsgi:application -w 2 -b :8000
    environment:
      DEBUG: ${DEBUG}
      DB_HOST: ${DB_HOST}
      DB_NAME: ${DB_NAME}
      DB_USER: ${DB_USER}
      DB_PORT: ${DB_PORT}
      DB_PASSWORD: ${DB_PASSWORD}
      SENDGRID_API_KEY: ${SENDGRID_API_KEY}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_STORAGE_BUCKET_NAME: ${AWS_STORAGE_BUCKET_NAME}
    depends_on:
      - postgres
      - redis
    expose:
      - "8000"
    volumes:
      - .:/writer-api
  redis:
    image: "redis:alpine"
  celery:
    build: .
    command: celery -A writer worker -l info
    volumes:
      - .:/writer-api
    depends_on:
      - postgres
      - redis
  celery-beat:
    build: .
    command: celery -A writer beat -l info
    volumes:
      - .:/writer-api
    depends_on:
      - postgres
      - redis
  nginx:
    restart: always
    build: ./nginx/
    ports:
      - "80:80"
    depends_on:
      - web
volumes:
  pgdata:

.gitlab-ci.yml

image: tmaier/docker-compose:latest

services:
  - docker:dind

before_script:
  - docker info
  - docker-compose --version

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script:
    - echo "Building the app"
    - docker-compose build

test:
  stage: test
  variables:
  script:
    - echo "Testing"
    - docker-compose run web coverage run manage.py test

deploy-staging:
  stage: deploy
  only:
    - develop
  script:
    - echo "Deploying staging"
    - docker-compose up -d

deploy-production:
  stage: deploy
  only:
    - master
  script:
    - echo "Deploying production"
    - docker-compose up -d

Here are my settings for my variables

Here is my failed pipeline job

Answer 1

The SECRET_KEY variable will be available to all your CI jobs, as configured. However, I don't see any references to it in your Docker Compose file to pass it to one or more of your services. For the web service to use it, you'd map it in like the other variables you already have.

  web:
    build: .
    command: /usr/local/bin/gunicorn writer.wsgi:application -w 2 -b :8000
    environment:
      SECRET_KEY: ${SECRET_KEY}
      DEBUG: ${DEBUG}
      …

As for creating the database, you should wrap up whatever you currently run interactively in the postgres container in a SQL file or shell script, and then bind-mount it into the container's initialization scripts directory under /docker-entrypoint-initdb.d . See the Initialization scripts section of the postgres image's description for more details.

Answer 2

Don't keep .env file inside your .gitignore. Please keep it in the root directory where the yml files are present. Once it is done, yml files will detect environment variables automatically while executing.

Answer 3

In my expirience the best way to pass enviroment variables to a container docker is creating an environment file, which works to development environment and production environment:

GitLab CI/CD variables

you must create an environment file on GitLab CI/CD, following the next steps, on your project GitLab you must go to:

settings > CI/CD > Variables

on that you must create a ENV_FILE

Demo image

next on your build stage in .gitlab-ci.yml copy the ENV_FILE to local process

.gitlab-ci.yml

build:
  stage: build
  script:
    - cp $ENV_FILE .env
    - echo "Building the app"
    - docker-compose build

your Dockerfile should be stay normally so it doesn't have to change

Dockerfile

FROM python:3.8.6-slim

# Rest of setup goes here...

COPY .env .env

Answer 4

In order for compose variable substitution to work, when the user is not added to docker group, you must add the -E flag to sudo.

script:
        - sudo -E docker-compose build