Why does variables substitution not work when using multiple env_files in docker-compose.yml?

Question

I'm trying to get a docker-compose file working with multiple .env files, and I'm not having any luck. I'm trying to setup three .env files:

• global settings that are the same across all container instances
• environment-specific settings (stuff just for test or dev)
• local settings - overridable things that a developer might need to change in case they have conflicts with, say, a port number

My docker-compose.yml file looks like this:

version: '2'
services:
  db:
    env_file:
      - ./.env
      - ./.env.${ENV}
      - ./.env.local
    image: postgres
    ports:
      - ${POSTGRES_PORT}:5432

.env looks like this:

POSTGRES_USER=myapp

and the .env.development looks like this:

POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=localhost
POSTGRES_PORT=25432
POSTGRES_DB=myapp_development

.env.local doesn't exist in this case.

After running ENV=development docker-compose up , I receive the following output:

$ ENV=development docker-compose up
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.db.ports is invalid: Invalid port ":5432", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]

From that error message, it looks like none of my environment variables are being used. I just upgraded to the newest available docker-compose as well - same errors:

$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6

Any ideas here? Would be nice to have a single docker-compose.yml that would work across multiple environments.

Answer 1

Keep in mind that there are 2 different environments where you are defining variables. The host machine where you are executing the docker-compose command, and the container itself (running the db service in your case).

Your docker-compose.yml file has access to your host's environment variables. Hence ENV is reachable from the docker-compose command, but not these in your .env files.

On the contrary, the value for ENV is not reachable inside the container, but all variables defined in your .env files will.

I don't know if you really need your db container to access the variables defined on your .env.development . But at least seem that your host machine needs to have the content of that file, so when the docker-compose command is called, the POSTGRES_PORT variable is defined.

To fix your specific problem you would need to define the environment variables on your host machine too, not only for the container. You could do something like this:

#Set for host
ENV=development

#Also sets the variables on the host 
source ./.env.$ENV

#POSTGRES_PORT defined in .env.development is used here
docker-compose up

#since env_file also contains .env.development, the variables will be reachable from the container.

Hope that helps.

Answer 2

In order to apply different/multiple env_files depending on the running environment, such as development/staging/production, I think a better way for docker-compose is to use multiple docker-compose yml files.

For example:

1. Start with a base file that defines the canonical configuration for the services.

docker-compose.yml

web:
  image: example/my_web_app:latest
  env_file:
    - .env

2. Add the override file for development, as its name implies, can contain configuration overrides for existing services or entirely new services.

docker-compose.override.yml

web:
  build: .
  volumes:
    - '.:/code'
  ports:
    - 8883:80
  env_file:
    - .env.dev

When you run docker-compose up it reads the overrides automatically.

3. Create another override file for the production environment.

docker-compose.prod.yml

web:
  ports:
    - 80:80
  env_file:
    - .env.prod

To deploy with this production Compose file you can run

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

Note

My Docker version:

$ docker -v
Docker version 18.06.1-ce, build e68fc7a

$ docker-compose -v
docker-compose version 1.22.0, build f46880fe

Reference: https://docs.docker.com/compose/extends/

Answer 3

while reading this page: https://docs.docker.com/compose/environment-variables/

and from my understanding, you should do the following:

for the global variables(that should not change) make an env file like so:

VAR1=VALUE1
VAR2=VALUE2

and for the others(that might change) you should add their name under environment in docker-compose.yml like this:

environment:
    - VAR1
    - VAR2

this will take the VAR1 and VAR2 values from the shell you are running docker-compose.

I hope this helps.

Answer 4

There is a misconception regarding the .env file and the env_file option in the docker-compose.yml, as it is very ambiguous. Shin points it out very nicely in the github issue docker-compose doesn't use env_file . I will just quote his summary:

• Variable substitution in your docker-compose.yml file will be pulled (in decreasing order of priority) from your shell's environment and your .env file.

• Variables available in your container are a combination of values found in your env_file files and values described in the environment section of the service.

• Those are two entirely separate sets of features.