Inter-pod communication in kubernetes

Question

I had previously created a pod with three containers namely - prometheus , blackbox-exporter and python-access-api . The blackbox-exporter runs on port 9115 and scrapes the targets generated by python-access-api container which is alerted in the prometheus for SSL Expiry certificate of the targets. Now I want to move the blackbox- exporter to a different pod. I have tried to establish this via service but I am failing to establish the communication between prometheus and blackbox-exporter now, since they are in a different pod. And as a result of this, I am unable to make probe for SSL Expiry certificate and hence, cannot see the alerts on prometheus . Below is the yaml file that I have used, can anyone please point out a way out of this problem. Please note that my configuration looks fine for prometheus and also the pods for blackbox and prometheus are running fine individually. Like I said above, I dont see they communicate with each other.

apiVersion: v1
kind: ReplicationController
metadata:
  name: blackbox-deployment
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    app: blackbox
  template:
    metadata:
      name: blackbox
      labels:
        app: blackbox
    spec:
      containers:
      - name: blackbox

Yaml file for Prometheus deployment

apiVersion: v1
kind: ReplicationController
metadata:
  name: python-daemon
  labels:
    app: prometheus-python
spec:
  replicas: 1
  selector:
    app: python
  template:
    metadata:
      name: python
      labels:
        app: python
    spec:
      containers:

The service that I have deployed:-

apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  selector:
    app: prometheus
  ports:
  - name: http
    port: 80
    targetPort: 9115
    protocol: TCP

The prometheus config is as follows

- job_name: blackbox
  params:
    module:
    - http_2xx
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  file_sd_configs:
  - files:
    - /var/suhas/targets.yml
    refresh_interval: 5m
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - source_labels: []
    separator: ;
    regex: (.*)
    target_label: __address__
    replacement: prometheus:9115
    action: replace

Answer 1

First of all, you would need to clarify which communications are taking place. Is prometheus accessing blackbox exporter via port 9115, or is blackbox exporter accessing prometheus via port 80? Depending on which is right, the service would be different.

In your service above, you are setting and endpoint which, when accessed via port 80, it will redirect traffic to port 9115 of your blackbox-exporter app. I will assume that prometheus is accessing the blackbox-exporter for the rest of the answer.

Would prometheus access blackbox exporter via port 9115 or port 80. It looks to me that, in your initial set up, the prometheus would use port 9115 to access. Therefore, there is no reason to change the port in the service to 80. Could you try setting port: 9115 in your service file instead?

Plus, make sure you configure prometheus to use the correct address. I assume that it would previously use 127.0.0.1:9115, now it would need to be prometheus:9115 (as you named the service prometheus , which can be bit confusing).