How to check if all docker-compose up services started successfully?

Question

How can I check if the services of docker-compose up started successfully?

Context: I'm adding CI to my project and as part of it, I'd like to have my CI build the docker image and execute a docker-compose command to see if all the services can be brought up successfully before moving on to the next phase of the CI.

I'm thinking that I should do something to the effect of:

$ docker build -t myimage .
$ docker-compose up -d

# attempt count the images with a state of "Exit" and if it's more than 1, return 1
# what happens if an image name contains the substring "Exit" but is running healthily?
$ if [[ $(docker-compose -f docker-compose-local.yml ps | grep "Exit" | wc -l) -ne 0 ]]; then return 1; fi

I suspect that looking for the state of the container through the ps subcommand isn't great and that there's a better solution. I've looked but haven't found one.

What are the possible states of a docker container? The docker docks don't mention the possible states.

Answer 1

You might be able to try running with a custom HEALTHCHECK command, which could write a status to a temporary file or something, which the CI process can check for, and then terminate the containers afterwards.

HEALTHCHECK will also cause the health status of each container to be stored in the respective health_status field in the output of docker inspect , which might be easier for you than docker ps .

More generally, I have worked on some applications where we created a type of test that we called a "smoke test" -- very similar to a full acceptance test, except that it could still use artificial or fixture data of some types. But the actual requests made to the web / container applications were real.

Then what you do is have a script which starts up your set of containers, followed by executing a test suite against whatever application(s) are backed by those containers (eg database queries, http requests, whatever).

Often you can re-use a unit testing framework from your language of choice for this. In my case, we used pytest to write some Python-based smoke tests ... launch all the containers, then hit them with a variety of artificial requests for which the correct response was known. If the Python test process exited successfully, then the overall test passed and it would bring down the containers.

I like this approach because then the definition of "check if all docker-compose up services started successfully" just becomes some automated test like any other, and it is checked into source code somewhere. If the applications change, you change the tests accordingly, or else they cause failing builds.

Whereas, if you used a simplistic HEALTHCHECK command, it's feasible you could get false information. Maybe the containers appear to start and run, but if a certain data resource, or remote connection, etc., didn't succeed at start-up, you won't know unless your HEALTHCHECK explicitly involves verifying it.

Really it rarely matters to know just that the containers are up and running. Instead, what usually matters is some suite of "real" tests that hit the running containers and prove that everything needed to operate in a real use-case scenario is actually working correctly.

Answer 2

How to check if all docker-compose up services started successfully?

You can check if all services in docker-compose are running, by comparing all running services and with all services. Note that this may not mean that your service is running - only that the docker container is running. The following shell script just compares the count of running and all services:

running="$(docker-compose ps --services --filter "status=running")"
services="$(docker-compose ps --services)"
if [ "$running" != "$services" ]; then
    echo "Following services are not running:" 
    # Bash specific
    comm -13 <(sort <<<"$running") <(sort <<<"$services")
else
    echo "All services are running"
fi

It seems that negating filter is not possible in docker, so it has to be that way.

Answer 3

Checking for .State.Status , .State.Running , etc. will tell you if it's running, but it's better to ensure that the health of your containers.

Below is a script that you can run that will make sure that all docker compose service containers are in good health before executing a command within one of the predefined service container. It exits with code 1 and prints out the docker logs if the wait time/attempts threshold has been reached.

Modified from npm sql-mdb .

    #!/bin/bash
    # Wait for all docker compose service healthchecks to be in a "healthy" state before executing a "docker exec -it $2_container_name bash $3"
    # Example:
    # ./this_script.sh "my_docker_compose_project" "mymainservice" "ls -al"
    ##############################################################################################################################
    # $1 Docker compose project name (required)
    # $2 Docker compose service name that will be used to run the execution command (required)
    # $3 The actual execution command that will be ran within the execution service container (required)
    # $4 The max number of attempts to wait for a healthy healthcheck on each service (optional, defaults to 80)
    # $5 The sleep time between attempts to wait for a healthy healthcheck on each service (optional, default to 2)

    attempts_max=$([[ -n "$4" ]] && echo "$4" || echo 80)
    sleep_time=$([[ -n "$5" ]] && echo "$5" || echo 2)
    printf "=======> Waiting for docker healthchecks &2

    if [[ -z "$1" ]]; then
      echo "Docker compose project name is required as the first passed argument!" >&2
      exit 1
    fi
    if [[ -z "$2" ]]; then
      echo "Docker compose service name for project \"$1\" used as the executing container is required as the second passed argument!" >&2
      exit 1
    fi
    if [[ -z "$3" ]]; then
      echo "The command that will be executed for the docker compose project \"$1\" within the container for service \"$2\" is required as the third passed argument!" >&2
      exit 1
    fi

    dc_services="$(docker-compose -p $1 ps --services)"
    if [[ -z "$dc_services" ]]; then
      echo "No docker compose services found for project: \"$1\"!" >&2
      exit 1
    fi

    dc_service_exec="$1_${2}_1"
    dc_service_cnt=$(wc -l &2; exit 1; }
        fi
        if [[ health == "healthy" ]]; then
          echo "Docker healthcheck on service $dc_service ($health)"
          dc_service_healthy_cnt=$(( $dc_service_healthy_cnt + 1 ))
          break
        fi
        sleep "$sleep_time"
      done
      if [[ $health != "healthy"  ]]; then
        echo "Failed to wait for docker healthcheck on \"$dc_service\" ($health) after $attempt attempts - services:\n$dc_services\n"
        docker logs --details "$dc_service"
        exit 1
      fi
      echo "SERVICE: $dc_service"
    done
    if (( "$dc_service_healthy_cnt" >= "$dc_service_cnt" )); then
      echo "Docker healthcheck are healthy on services: \"$dc_services\" - Executing \"$3\" in container \"$dc_service_exec\""
      docker exec -it "$dc_service_exec" bash -c "$3"
      [[ $? != 0 ]] && { echo "Failed to execute \"$3\" in docker container \"$dc_service_exec\"" >&2; exit 1; }
    fi