将在负载均衡器后面运行的 ec2 实例注册为普罗米修斯中的目标时出现的问题

Question

My Setup I have setup prometheus and grafana on a machine - M1 I have a AWS Load balancer (LB1) which is forwarding the requests to app instances and handling the ssl part. Load Balancer has one listener listening on https:443 and is forwarding requests to Target group(TG1) which has 2 EC2 instances (I1, I2). Each EC2 instance is also running nginx.

Request lifecycle : Browser requests -> https://api.example.com

1. Request goes to load balancer.
2. Load balancer redirects it to one of the instances in TG1
3. Nginx running on instance further forwards the request to one of the app workers.

The problem is coming when i am trying to fetch metrics from prometheus using node exporter from each of the instances(I1 & I2). I have installed the node exporter in each of the instances which is running. How do I register Instance I1 & Instance I2 as targets in prometheus.yml file?

I have tries 2 approaches:

Approach 1:

Using load balancer -

I simply installed node exporter in each of the instances and running node exporter on port 9100. Incoming requests to the machine from load balancer will be forwarded to localhost:9100/metrics by nginx.

Example prometheus.yml file:

  - job_name: "node_exporter"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    scheme: "https"
    metrics_path: "/node-exporter/metrics"
    static_configs:
      - targets: ["api.example.com:443"]
    basic_auth:
      username: 'username'
      password: 'password'

The problem here is it registers just one host as the target. I was expecting to see two hosts on prometheus with the same name.

Question:

1. Is there a way I can forward requests to a unique instance in loadbalancer target group based on port? This way, i can run node exporter on different ports on different instances and specify which instance to forward the request to based on port.
2. Is there a way I can forward requests to a unique instance in loadbalancer target group based on ip address in url? This way, i can run node exporter on same ports on different instances and just forward the request to individual machine based on the ip address in the url.

Approach 2:

Using IP address -

I registered the IP address as a prometheus target and opened the ports on machines.

Example prometheus.yml file:

  - job_name: "node_exporter"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    scheme: "http"
    metrics_path: "/node-exporter/metrics"
    static_configs:
      - targets: ["162.x.y.1:9100", "162.x.y.2:9100"]
    basic_auth:
      username: 'username'
      password: 'password'

The problem here how do i make it work with https scheme?

Question:

1. How do i enable ssl for an ip address and for sending requests to port other than 443? Is it possible?

Summary:

What is the best way to set up prometheus to start gathering metrics from machines running behind AWS Load balancer in a single target group and accessible via common domain name? How do i register the prometheus targets in this case?

Answer 1

I was finally able to figure it out. As long as things were on the HTTP scheme, everything worked great. I was facing challenges when I tried to use HTTPS scheme for multiple targets under a job.

Since I have 2 EC2 instances running behind a load balancer, so I don't have DNS names for them. I had to work with the IP addresses.

Finally, I was able to make it work using self-signed SSL certificates which I used for NGINX and configuring Prometheus to ignore SSL verification .

Now, the only thing is that I have to do this on each of the instances that I want to monitor.

My prometheus.yml file looks something like this:

- job_name: "node_exporter"
  scheme: "https"
  static_configs:
    - targets: ["192.x.y.z:443"]
  tls_config:
    insecure_skip_verify: true
  basic_auth:
    username: 'username'
    password: 'password'