Expose package metrics to Prometheus with prometheus_client

Question

I have two files running the following code on their own:

sandbox1.py

from prometheus_client import Counter
import time
while True:
    my_counter1 = Counter('my_counter1', 'My counter)
    my_counter1.inc()
    time.sleep(1)

sandbox2.py

from prometheus_client import Counter
import time
while True:
    my_counter2 = Counter('my_counter2', 'My counter)
    my_counter2.inc()
    time.sleep(2)

Is there a way of using prometheus_client to expose my_counter1 and my_counter2 to my Prometheus server?

I've had a look at https://github.com/prometheus/client_python , but I'm not too sure how how to call the metrics from sandbox1.py and sandbox2.py into the file that actually exposes the metrics to Prometheus. Do the metrics always have to be part of the same file?

Answer 1

Prometheus server scrapes HTTP endpoints that provide metrics. This differs from some other metric systems where the metrics are pushed to the metric system. Because Prometheus scrapes metric endpoints, you need to do two things:

1. Expose the metrics from your client using an HTTP server|endpoint
2. Configure the Prometheus server to target this metric server|endpoint

If you run the Three Step Demo example on the page that you reference, and then browse http://localhost:8000, you should see something like:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 357.0
python_gc_objects_collected_total{generation="1"} 0.0
python_gc_objects_collected_total{generation="2"} 0.0
...
...
# HELP request_processing_seconds Time spent processing request
# TYPE request_processing_seconds summary
request_processing_seconds_count 4.0
request_processing_seconds_sum 2.0374009040533565
# HELP request_processing_seconds_created Time spent processing request
# TYPE request_processing_seconds_created gauge
request_processing_seconds_created 1.6004497426536365e+09

This is the page that your Prometheus server would be configured to scrape. You can see, eg python_gc_objects_collected_total counter is provided. Prometheus metrics are human readable which is useful.

If you combine, your sandbox1.py into this example:

from prometheus_client import start_http_server, Counter, Summary
import random
import time

# Create a metric to track time spent and requests made.
REQUEST_TIME = Summary('request_processing_seconds',
                       'Time spent processing request')


# Decorate function with metric.
@REQUEST_TIME.time()
def process_request(t):
    """A dummy function that takes some time."""
    time.sleep(t)


if __name__ == '__main__':
    my_counter1 = Counter('my_counter1', 'My counter')
    # Start up the server to expose the metrics.
    start_http_server(8000)
    # Generate some requests.
    while True:
        my_counter1.inc()
        process_request(random.random())

And run the code again, you should now see:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 357.0
python_gc_objects_collected_total{generation="1"} 0.0
python_gc_objects_collected_total{generation="2"} 0.0
...
...
# HELP my_counter1_total My counter
# TYPE my_counter1_total counter
my_counter1_total 13.0
# HELP my_counter1_created My counter
# TYPE my_counter1_created gauge
my_counter1_created 1.6004498408280134e+09

NOTE At the bottom of the page is my_counter1_total and my_counter1_created which correspond to your my_counter1 = Counter("my_counter1","My counter")

If you point a Prometheus server at this target ( localhost:8000 ), you should be able to eg graph the my_counter1_total counter and the my_counter1_created gauge.

Create a file called prometheus.yml :

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Self
  - job_name: "prometheus-server"
    static_configs:
      - targets:
          - "localhost:9090"

  # Python example
  - job_name: "63957470"
    static_configs:
      - targets:
          - "localhost:8000"

And then run Prometheus:

docker run \
--interactive --tty \
--net=host \
--volume=${PWD}/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus@sha256:f3ada803723ccbc443ebea19f7ab24d3323def496e222134bf9ed54ae5b787bd

NOTE This assumes that prometheus.yml is in the working directory

The, you can browse Prometheus on http://localhost:9090

You can see your code configured as a target: http://localhost:9090/targets

And you can query|graph your metrics. Type eg my_counter1_ to see both and then my_counter1_total for the Counter: