Find url/ip of container running in docker-compose in gitlab ci

Question

I have an application that runs in docker-compose (for acceptance testing). The acceptance tests work locally, but they require the host (or ip) of the webservice container running in docker-compose in order to send requests to it. This works fine locally, but I cannot find the ip of the container when it is running in a gitlab ci server. I've tried the following few solutions (all of which work when running locally, but none of which work in gitlab ci) to find the url of the container running in docker-compose in gitlab ci server:

1. use "docker" as the host. This works for an application running in docker, but not docker-compose
2. use docker-inspect to find the ip of the container ( docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' reading-comprehension )
3. assign a static ip to the container using a network in docker-compose.yml (latest attempt).

The gitlab ci file can be found here: https://gitlab.com/connorbutch/reading-comprehension/-/blob/9-list-all-assessments/.gitlab-ci.yml

    image: connorbutch/gradle-and-java-11:alpha

variables:
  GRADLE_OPTS: "-Dorg.gradle.daemon=false"
  DOCKER_HOST: "tcp://docker:2375"
  DOCKER_DRIVER: "overlay2"

before_script:
  - export GRADLE_USER_HOME=`pwd`/.gradle

services:
  - docker:stable-dind

stages:
  - build
  - docker_build
  - acceptance_test

unit_test:
  stage: build
  script: ./gradlew check
  cache:
    key: "$CI_COMMIT_REF_NAME"
    policy: pull
    paths:
      - build
      - .gradle

build:
  stage: build
  script:
    - ./gradlew clean quarkusBuild
    - ./gradlew clean build -Dquarkus.package.type=native -Dquarkus.native.container-build=true
  cache:
    key: "$CI_COMMIT_REF_NAME"
    policy: push
    paths:
      - build
      - .gradle
  artifacts:
    paths:
      - reading-comprehension-server-quarkus-impl/build/

docker_build:
  stage: docker_build
  script:
    - cd reading-comprehension-server-quarkus-impl
    - docker build -f infrastructure/Dockerfile -t registry.gitlab.com/connorbutch/reading-comprehension:$CI_COMMIT_SHORT_SHA  .
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker push registry.gitlab.com/connorbutch/reading-comprehension:$CI_COMMIT_SHORT_SHA

acceptance_test:
  stage: acceptance_test
  only:
    - merge_requests
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - cd reading-comprehension-server-quarkus-impl/infrastructure
    - export IMAGE_TAG=$CI_COMMIT_SHORT_SHA
    - docker-compose up -d & ../../wait-for-it-2.sh
    - cd ../..
    - ./gradlew -DBASE_URL='192.168.0.8' acceptanceTest
  artifacts:
    paths:
      - reading-comprehension/reading-comprehension-server-quarkus-impl/build/

The docker-compose file can be found here: https://gitlab.com/connorbutch/reading-comprehension/-/blob/9-list-all-assessments/reading-comprehension-server-quarkus-impl/infrastructure/docker-compose.yml

Find the output of one of the failed jobs here: https://gitlab.com/connorbutch/reading-comprehension/-/jobs/734771859

#This file is NOT ever intended for use in production.  Docker-compose is a great tool for running
#database with our application for acceptance testing.
version: '3.3'

networks:
  network:
    ipam:
      driver: default
      config:
        - subnet: 192.168.0.0/24

services:
  db:
    image: mysql:5.7.10
    container_name: "db"
    restart: always
    environment:
       MYSQL_DATABASE: "rc"
       MYSQL_USER: "user"
       MYSQL_PASSWORD: "password"
       MYSQL_ROOT_PASSWORD: "password"
       MYSQL_ROOT_HOST: "%"
    networks:
      network:
        ipv4_address: 192.168.0.4
    ports:
      - '3306:3306'
    expose:
      - '3306'
    volumes:
      - db:/var/lib/mysql
  reading-comprehension-ws:
    image: "registry.gitlab.com/connorbutch/reading-comprehension:${IMAGE_TAG}"
    container_name: "reading-comprehension"
    restart: on-failure
    environment:
      WAIT_HOSTS: "db:3306"
      DB_USER: "user"
      DB_PASSWORD: "password"
      DB_JDBC_URL: "jdbc:mysql://192.168.0.4:3306/rc"
    networks:
      network:
        ipv4_address: 192.168.0.8
    ports:
      - 8080:8080
    expose:
      - 8080
volumes:
  db:

Does anyone have any idea on how to access the ip of the container running in docker-compose on gitlab ci server? Any suggestions are welcome.

Thanks,

Connor

Answer 1

This is little bit tricky, just few days ago I had similar problem but with VPN from CI to client :)

EDIT: Solution for on-premise gitlab instances

Create custom network for gitlab runners:

docker network create --subnet=172.16.0.0/28 \
 --opt com.docker.network.bridge.name=gitlab-runners \
 --opt com.docker.network.bridge.enable_icc=true \
 --opt com.docker.network.bridge.enable_ip_masquerade=true \
 --opt com.docker.network.bridge.host_binding_ipv4=0.0.0.0 \
 --opt com.docker.network.driver.mtu=9001 gitlab-runners

Attach new network to gitlab-runners

# /etc/gitlab-runner/config.toml
[[runners]]
....
   [runners.docker]
   ....
   network_mode = "gitlab-runners"

Restart runners.

And finally gitlab-ci.yml

start-vpn:
    stage: prepare-deploy
    image: docker:stable
    cache: {}
    variables:
        GIT_STRATEGY: none
    script:
        - >
            docker run -it -d --rm
            --name vpn-branch-$CI_COMMIT_REF_NAME
            --privileged
            --net gitlab-runners
            -e VPNADDR=$VPN_SERVER
            -e VPNUSER=$VPN_USER
            -e VPNPASS=$VPN_PASSWORD
            auchandirect/forticlient || true && sleep 2
        - > 
            docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 
            vpn-branch-$CI_COMMIT_REF_NAME > vpn_container_ip
    artifacts:
        paths:
        - vpn_container_ip

And in next step you can use something like:

before_script:
        - ip route add 10.230.163.0/24 via $(cat vpn_container_ip) # prod/dev
        - ip route add 10.230.164.0/24 via $(cat vpn_container_ip) # test

EDIT: Solution for gitlab.com

Base on gitlab issue answer port mapping in DinD is bit different from nonDinD gitlab-runner and for exposed ports you should use hostname 'docker'.

Example:

services:
  - docker:stable-dind

variables:
  DOCKER_HOST: "tcp://docker:2375"

stages:
  - test

test env:
  image: tmaier/docker-compose:latest
  stage: test
  script:
    # containous/whoami with exposed port 80:80
    - docker-compose up -d
    - apk --no-cache add curl
    - curl docker:80              # <------- 
    - docker-compose down

Answer 2

I'm using docker and not docker-compose and the solution above doesn't work for me/ I am using my own image based on node in which I install docker & buildx like this:

ARG NODE_VER=lts-alpine

FROM node:${NODE_VER}

ARG BUILDX_VERSION=0.5.1
ARG DOCKER_VERSION=20.10.6
ARG BUILDX_ARCH=linux-arm-v7

RUN apk --no-cache add curl

# install docker
RUN curl -SL "https://download.docker.com/linux/static/stable/armhf/docker-${DOCKER_VERSION}.tgz" | \
        tar -xz --strip-components 1 --directory /usr/local/bin/
COPY docker/modprobe.sh /usr/local/bin/modprobe
# replace node entrypoint by docker one /!\
COPY docker/docker-entrypoint.sh /usr/local/bin/
ENV DOCKER_TLS_CERTDIR=/certs
RUN mkdir /certs /certs/client && chmod 1777 /certs /certs/client

# download buildx
RUN mkdir -p /usr/lib/docker/cli-plugins \
 && curl -L \
      --output /usr/lib/docker/cli-plugins/docker-buildx \
      "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.${BUILDX_ARCH}"
RUN chmod a+x /usr/lib/docker/cli-plugins/docker-buildx
RUN mkdir -p /etc/docker && echo '{"experimental": true}' > /usr/lib/docker/config.json

My gitlab-ci.yml contains:

image: myimageabove

variables:
  DOCKER_DRIVER: overlay2
  PLATFORMS: linux/arm/v7
  IMAGE_NAME: ${CI_PROJECT_NAME}
  TAG: ${CI_COMMIT_BRANCH}-latest
  REGISTRY: registry.gitlab.com
  REGISTRY_ROOT: mygroup
  WEBSOCKETD_VER: 0.4.1
#  DOCKER_GATEWAY_HOST: 172.17.0.1
  DOCKER_GATEWAY_HOST: docker

services:
  - docker:dind

before_script:
  - docker info

build:
  stage: build
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" ${REGISTRY}
    - docker buildx create --use
    - docker buildx build --platform $PLATFORMS --tag "${REGISTRY}/${REGISTRY_ROOT}/${IMAGE_NAME}:${TAG}" --push .

test:
  stage: test
  variables:
    WSD_DIR: /tmp/websocketd
    WSD_FILE: /tmp/websocketd/websocketd
  cache:
    key: websocketd
    paths: 
      - ${WSD_DIR}
  before_script:
# download websocketd and put in cache if needed
    - if [ ! -f ${WSD_FILE} ]; then
        mkdir -p ${WSD_DIR};
        curl -o ${WSD_FILE}.zip -L "https://github.com/joewalnes/websocketd/releases/download/v${WEBSOCKETD_VER}/websocketd-${WEBSOCKETD_VER}-linux_arm.zip";
        unzip -o ${WSD_FILE}.zip websocketd -d ${WSD_DIR};
        chmod 755 ${WSD_FILE};
      fi;
    - mkdir /home/pier
    - cp -R ./test/resources/* /home/pier
    # get websocketd from cache
    - cp ${WSD_FILE} /home/pier/Admin/websocketd
    # setup envt variables
    - JWT_KEY=$(cat /home/pier/Services/Secrets/WEBSOCKETD_KEY)
    # - DOCKER_GATEWAY_HOST=$(/sbin/ip route|awk '/default/ { print $3 }')
    # - DOCKER_GATEWAY_HOST=$(hostname)
    - ENVT="-e BASE_URL=/ -e JWT_KEY=$JWT_KEY -e WEBSOCKETD_KEY=$JWT_KEY -e WEBSOCKET_URL=ws://${DOCKER_GATEWAY_HOST:-host.docker.internal}:8088 -e SERVICES_DIR=/home/pier/Services"
    - VOLUMES='-v /tmp:/config -v /home/pier/Services:/services -v /etc/wireguard:/etc/wireguard'

  script:
    # start websocketd
    - /home/pier/start.sh &
    # start docker pier admin
    - docker run -p 4000:4000 ${ENVT} ${VOLUMES} ${REGISTRY}/${REGISTRY_ROOT}/${IMAGE_NAME}:${TAG}
    # run postman tests
    - newman run https://api.getpostman.com/collections/${POSTMAN_COLLECTION_UID}?apikey=${POSTMAN_API_KEY}

deploy:
  stage: deploy
  script:
    # just push to docker hub
    - docker login -u "$DOCKERHUB_REGISTRY_USER" -p "$DOCKERHUB_REGISTRY_PASSWORD" ${DOCKERHUB}
    - docker buildx build --platform $PLATFORMS --tag "${DOCKERHUB}/mygroup/${IMAGE}:${TAG}" --push .

When I run this, the build job works alright, then the test "before_script" works but when the script starts, I get the following trace:

# <= this starts the websocketd server locally on port 8088 =>
$ /home/pier/start.sh &
# <= this starts the image I just built which should connect to the above websocketd server =>
$ docker run -p 4000:4000 ${ENVT} ${VOLUMES} ${REGISTRY}/${REGISTRY_ROOT}/${IMAGE_NAME}:${TAG}     
# <= trace of the websocketd server start with url ws://runner-hbghjvzp-project-22314059-concurrent-0:8088/ =>
Tue, 11 May 2021 12:08:13 +0000 | INFO   | server     |  | Serving using application   : ./websocket-server.py 
Tue, 11 May 2021 12:08:13 +0000 | INFO   | server     |  | Starting WebSocket server   : ws://runner-hbghjvzp-project-22314059-concurrent-0:8088/
# <= trace of the image start saying it tires to conenct to the websocketd server
Websocket connecting to ws://docker:8088 ...
Listen on port 4000
# <= trace with ENOTFOUND on "docker" address =>
websocket connection failed: Error: getaddrinfo ENOTFOUND docker
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'docker'
}
/pier/storage/websocket-client.js:52
            throw err;
            ^
Error: getaddrinfo ENOTFOUND docker
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'docker'
}
Cleaning up file based variables 00:01
ERROR: Job failed: exit code 1
 

I tried other ways like:

websocket connection failed: Error: getaddrinfo ENOTFOUND host.docker.internal
websocket connection failed: Error: connect ETIMEDOUT 172.17.0.1:8088 # <= same error when trying $(/sbin/ip route|awk '/default/ { print $3 }') =>
websocket connection failed: Error: getaddrinfo ENOTFOUND runner-meuessxe-project-22314059-concurrent-0  # using $(hostname)

Out of new idea... Would greatly appreciate any help on that one.