Prometheus pushgateway simple metric monitor

Question

Im trying to learn prometheus (and eventually use it grafana). As of now my primary usecase that I am trying to learn is be able to update from my client (not have prometheus scrape). I have a pushgateway and I want my scripts/process to be able to update the prometheus (then grafana get that stats). I have a simple script to that I am starting out with, where I have get the cpu utilization (compute it as as a percentage) and send it prometheus:

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway, Summary, Histogram
from time import sleep
from random import randint, random

registry = CollectorRegistry()

cpu_util_sum_metric = Summary('cpu_util_summary', 'cpu_util_summary', registry=registry)
cpu_util_hist_metric = Summary('cpu_util_hist', 'cpu_util_hist', registry=registry)

for i in range(90):
  cpu_util = randint(0, 100)

  cpu_util_sum_metric.observe(float(cpu_util))
  cpu_util_hist_metric.observe(float(cpu_util))
  print('cpu util is: {}'.format(cpu_util))
  res = push_to_gateway('localhost:9091', job='cpu_stats', registry=registry)
  print('push_to_gateway result is:', str(res))
  sleep(5)

When I run this, in the prometheus browser I see the following metrics (based on my 'cpu_util_hist' and 'cpu_util_sum'):

• cpu_util_hist_count
• cpu_util_hist_created
• cpu_util_hist_sum
• cpu_util_summary_count
• cpu_util_summary_created
• cpu_util_summary_sum

First I figured I would have single points at different timestamps (kind of like in excel) of 'cpu_util_hist' or 'cpu_util_summary', that I can use to plot in prometheus or grafana. How can I acheive that? Do I need to have them as sum and calculate the rate/irate? What am I misunderstanding with prometheus.

Answer 1

I'll tell you how I think this works and we'll let others correct me ;-)

Prometheus 'prefers' to pull metrics from an endpoint. This is its inate behavior and for good reasons

The Pushgateway exists because not every system can itself function as a scrape target for Prometheus to have its metrics pulled.. In these situations, the Pushgateway functions as a proxy scrape target for the application; the application sends it metrics to Pushgateway, Prometheus scrapes Pushgateway.

When Prometheus scrapes an endpoint, this becomes the timestamp for the measurements that are acquired at that time. In the case of the Pushgateway, the timestampt is Pushgateway scrape time not proxied application time.

Metrics don't carry timestamps.

Prometheus builds time-series from metrics acquired through these scrapes (pulling directly from an app's metrics endpoint or pulling directly from a Pushgateway endpoint proxying metrics fro the app) with a frequency determined by how often the scrapes are made.

I think the answer to your question is that, over time, if Prometheus is configured to scrape the Pushgateway endpoint that's receiving your code's pushed metrics, it will be building time-series of the metrics for you to query|graph. Prometheus includes its own querying|graphing functionality that you may use to test this hypothesis and prove this to yourself.

Once you're confident that this is how it's working, you can then use Grafana against your Prometheus time-series source(s) to graph the data using Grafana instead.