Prometheus with multi-container pod on kubernetes

Question

I have a multi-container pod in my kubernetes deployment:

• java
• redis
• nginx

For every of those containers, there's a container with Prometheus exporter as well.

The question is how can I expose those ports to Prometheus if annotations section supports only one port per pod?

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port: 'xxxx'

but I need something like this:

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port_1: 'xxxx'
  prometheus.io/port_2: 'yyyy'
  prometheus.io/port_3: 'zzzz'

Maybe there's some other method to scrape all metrics from my multi-container pods? Thanks in advance for any kind of help.

Answer 1

Here's an example job for Prometheus. Put it in your own config.

Next, add:

annotations:
   prometheus.io/scrape: 'true'

to your pod metadata.

And on every container, which provides /metrics to prom, create an appropriate port, named metrics .

That's it. Prometheus would scrape only those ports, and there would be no situation, like when your redis instance would get http requests on its 6379 port.

Answer 2

The annotations you propose should work. Create one scrape_config per port annotation, keeping only targets matching the corresponding annotation port name.