Docker container ports not available right away

Question

I have Jenkins to build my project and run my unit test and integration test. Before all this, Jenkins is launching several docker containers (with mongodb , cassandra etc inside) and then it starts. Sometimes, my tests fail because the they cannot reach the resources. After digging a bit I noticed that not all my containers start. So here is what I have:

docker-compose.yml with the definition of each docker container

start_docker.sh script:

    time docker-compose pull
    time docker-compose --project-name $JOB_NAME up -d
    time docker-compose --project-name $JOB_NAME ps
    echo "Wait services are started"

    docker-compose --project-name $JOB_NAME ps -q
    container_names=`docker-compose --project-name $JOB_NAME ps -q`
    container_nb="${#container_names}"
    for container_name in $container_names; do

        ports=`docker port $container_name | cut -d "/" -f1 `
        service_ip=`docker inspect $container_name | grep "IPAddress" | grep 172 | sed "s/[^0-9]*\\([0-9\\.]\\+\\)[^0-9]*/\\1/"`

        for port in $ports; do
            while ! nc -z $service_ip $port; do
                sleep 1 # wait for 1 second before check again
            done
        done
    done

What I've noticed is that sometimes the ports variable is empty for some containers and if I'm displaying them here is what I see:

[unit_test] target2sellcoredevelop_dse_1                     /etc/dse/run.sh                  Up                             
[unit_test] target2sellcoredevelop_mongo_1                   docker-entrypoint.sh mongo ...   Up                             
[unit_test] target2sellcoredevelop_pentaho-pdi_1             /bin/sh -c /bin/bash -c "/ ...   Up      0.0.0.0:9999->9999/tcp 
[unit_test] target2sellcoredevelop_rabbitmq_1                docker-entrypoint.sh /init.sh    Up                             
[unit_test] target2sellcoredevelop_redis_sentinel1_1         docker-entrypoint.sh sh -c ...   Up                             
[unit_test] target2sellcoredevelop_rediscachereco_master_1   docker-entrypoint.sh redis ...   Up                             
[unit_test] target2sellcoredevelop_rediscatalog_master_1     docker-entrypoint.sh redis ...   Up                             
[unit_test] target2sellcoredevelop_redisuser_master_1        docker-entrypoint.sh redis ...   Up

As you can see some of them already have the port number and some of them don't. I have 2 problems:

1. Why the port number is not available? Is because some docker zombie process? If so, how I can clean the up?

2. How can I properly wait for all them to really start?

EDIT: I think my problem is that sometimes some containers does not start. My question: how do I investigate that ?

Answer 1

You have a number of different questions, but in Docker Compose, you can wait for certain containers to come up before others using dadarek/wait-for-dependencies .

1). Add a new service to your docker-compose.yml

  waitfordb:
    image: dadarek/wait-for-dependencies
    depends_on:
      - mongodb 
    command: mongodb:27017

2). Add the following config to the service that requires MongoDB to be up. This will essentially wait for MongoDB to be fully up.

depends_on: 
  - waitfordb

3). Startup compose

docker-compose run --rm waitfordb
docker-compose up -d <SERVICE_1> <SERVICE_2>

This may or may not resolve your other issues, but might be a good place to start.