Unable to bring up prometheus using docker-compose file. Always throws the error /etc/prometheus/prometheus.yaml: no such file or directory

Question

I have been trying to bring up a Prometheus container using docker-compose file. I have looked into the various solutions available online and none of them seem to work. Please go through my prometheus.yaml file and the docker-compose.yml file and let me know, what have I configured wrongly. My prometheus.yaml file is located at /root/prometheus/prometheus.yaml

Note: I'm trying to run the prometheus in the agent-mode and I'm running the docker-compose file from the /root path.

Error generated:

-bash-5.0# docker-compose up
[...]
prometheus    | ts=2022-05-12T14:28:25.350Z caller=main.go:447 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yaml)" file=/etc/prometheus/prometheus.yaml err="open /etc/prometheus/prometheus.yaml: no such file or directory"
prometheus exited with code 2

docker-compose.yml:

version: '3'
volumes:
  prometheus_data:
services:
  prometheus:
    image: prom/prometheus:v2.35.0
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yaml'
      - '--storage.agent.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--enable-feature=agent'
    expose:
      - 9090
    ports:
      - "9090:9090"

Update 1: Adding the directory tree structure below

-bash-5.0# pwd
/root
-bash-5.0# tree
.
|-- cadvisor
|-- docker-compose.yml
|-- docker-compose_1.yml
|-- prometheus
|   |-- prometheus.yaml
|   |-- prometheus.yml
|   |-- prometheus_old.yaml
|   `-- prometheus_old.yml
|-- prometheus.yaml
`-- prometheus.yml

1 directory, 9 files
-bash-5.0#

Update 2: I did some debugging and found out that the directory is being mounted, whereas the files are being mounted as directory.

Basically what I did was I made changes to the docker-compose.yml file as follows.

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.35.0
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus-test/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      #- '--storage.agent.path=/prometheus'
      - '--web.enable-lifecycle'
      #- '--enable-feature=agent'
    expose:
      - 9090
    ports:
      - "9090:9090"
volumes:
  prometheus_data:

In the above docker file I'm mounting my prometheus.yml file to a different location and let the prometheus configure with default config file present. Later I logged into the container and checked for the mounted files and this is what was seen.

-bash-5.0# docker container exec -it prometheus sh
/prometheus $ cd /etc
/etc $ ls -ltr
total 68
-rw-r--r--    1 root     root         18774 Feb 10  2019 services
-rw-r--r--    1 root     root           494 Aug 16  2019 nsswitch.conf
-rw-r--r--    1 root     root           118 Mar 22 21:07 localtime
-rw-r--r--    1 root     root           340 Apr 11 21:49 passwd
-rw-rw-r--    1 root     root           306 Apr 11 21:49 group
-rw-------    1 root     root           136 Apr 13 00:25 shadow
drwxr-xr-x    6 root     root          4096 Apr 13 00:25 network
drwxr-xr-x    3 root     root          4096 Apr 15 10:54 ssl
lrwxrwxrwx    1 root     root            12 May 13 10:54 mtab -> /proc/mounts
-rw-r--r--    1 root     root           174 May 13 10:54 hosts
-rw-r--r--    1 root     root            13 May 13 10:54 hostname
-rw-r--r--    1 root     root            38 May 13 10:54 resolv.conf
drwxr-xr-x    3 root     root          4096 May 13 11:00 prometheus-test
drwxr-xr-x    1 nobody   nobody        4096 May 13 11:00 prometheus
/etc $
/etc $ cd prometheus-test/
/etc/prometheus-test $ ls
prometheus.yml   
/etc/prometheus-test $ ls -ltr
total 8
drwxr-xr-x    2 root     root          4096 May 13 10:54 prometheus.yml
/etc/prometheus-test $

From the above we can observe that the prometheus.yml file is being mounted as a directory instead of a file. Can anyone please let me know about this.

OS: Ubuntu 20.04 This is a vm instance running on the ESXI server

-bash-5.0# docker version
Client: Docker Engine - Community
 Version:           19.03.10
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        9424aeaee9
 Built:             Thu May 28 22:16:52 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       633a0ea838f10e000b7c6d6eed1623e6e988b5bb
  Built:            Sat May  9 16:43:52 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.2
  GitCommit:        ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:
-bash-5.0#
-bash-5.0#
-bash-5.0#
-bash-5.0# docker context ls
NAME                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm
-bash-5.0#

Answer 1

I think you may be over-complicating things. Given this docker-compose.yaml :

version: '3'
volumes:
  prometheus_data:
services:
  prometheus:
    image: prom/prometheus:v2.35.0
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.agent.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--enable-feature=agent'

And this directory structure:

$ tree .
.
├── docker-compose.yaml
└── prometheus
    └── prometheus.yml

It Just Works:

$ docker-compose up
Starting prometheus ... done
Attaching to prometheus
prometheus    | ts=2022-05-13T12:12:16.206Z caller=main.go:187 level=info msg="Experimental agent mode enabled."
prometheus    | ts=2022-05-13T12:12:16.207Z caller=main.go:525 level=info msg="Starting Prometheus" version="(version=2.35.0, branch=HEAD, revision=6656cd29fe6ac92bab91ecec0fe162ef0f187654)"
prometheus    | ts=2022-05-13T12:12:16.207Z caller=main.go:530 level=info build_context="(go=go1.18.1, user=root@cf6852b14d68, date=20220421-09:53:42)"
prometheus    | ts=2022-05-13T12:12:16.207Z caller=main.go:531 level=info host_details="(Linux 5.17.5-100.fc34.x86_64 #1 SMP PREEMPT Thu Apr 28 16:02:54 UTC 2022 x86_64 02da15afa8e7 (none))"
prometheus    | ts=2022-05-13T12:12:16.207Z caller=main.go:532 level=info fd_limits="(soft=1073741816, hard=1073741816)"
prometheus    | ts=2022-05-13T12:12:16.207Z caller=main.go:533 level=info vm_limits="(soft=unlimited, hard=unlimited)"
prometheus    | ts=2022-05-13T12:12:16.208Z caller=web.go:541 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
prometheus    | ts=2022-05-13T12:12:16.208Z caller=main.go:1013 level=info msg="Starting WAL storage ..."
prometheus    | ts=2022-05-13T12:12:16.212Z caller=db.go:332 level=info msg="replaying WAL, this may take a while" dir=/prometheus/wal
prometheus    | ts=2022-05-13T12:12:16.213Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
prometheus    | ts=2022-05-13T12:12:16.214Z caller=db.go:383 level=info msg="WAL segment loaded" segment=0 maxSegment=1
prometheus    | ts=2022-05-13T12:12:16.214Z caller=db.go:383 level=info msg="WAL segment loaded" segment=1 maxSegment=1
prometheus    | ts=2022-05-13T12:12:16.215Z caller=main.go:1034 level=info fs_type=XFS_SUPER_MAGIC
prometheus    | ts=2022-05-13T12:12:16.215Z caller=main.go:1037 level=info msg="Agent WAL storage started"
prometheus    | ts=2022-05-13T12:12:16.215Z caller=main.go:1162 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus    | ts=2022-05-13T12:12:16.216Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Starting WAL watcher" queue=b4f547
prometheus    | ts=2022-05-13T12:12:16.216Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Starting scraped metadata watcher"
prometheus    | ts=2022-05-13T12:12:16.216Z caller=main.go:1199 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=809.925µs db_storage=275ns remote_storage=282.833µs web_handler=394ns query_engine=377ns scrape=211.707µs scrape_sd=43.033µs notify=609ns notify_sd=911ns rules=143ns tracing=1.98µs
prometheus    | ts=2022-05-13T12:12:16.216Z caller=main.go:930 level=info msg="Server is ready to receive web requests."
prometheus    | ts=2022-05-13T12:12:16.216Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Replaying WAL" queue=b4f547
prometheus    | ts=2022-05-13T12:12:22.558Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Done replaying WAL" duration=6.341973747s

---

If this fails, the first diagnostic step is probably to remove the command section from docker-compose.yaml and add an entrypoint like this:

version: '3'
volumes:
  prometheus_data:
services:
  prometheus:
    image: prom/prometheus:v2.35.0
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    entrypoint:
      - sleep
      - inf

This will come up and just run sleep , allowing you to docker-compose exec into the container and explore the filesystem. If you find that /etc/prometheus is empty, that suggests that the you're not running docker-compose on the same system as the docker daemon (so when it attempts to reference a host path, like ./prometheus , it doesn't find it).