How to collect prometheus metrics from multiple python-flask sub-process?

Question

I have main() function which spawns two separate sub-processes. These both sub-process shares metrics. How can I share metrics for both process and keep it updating? Here, is my snippet for more understanding.

from multiprocessing import Process
import prometheus_client as prom
from prometheus_client import Counter

# Metrics
c1 = prom.gauge('Counter1', 'Number of Request')
c2 = prom.gauge('Gauge1', 'Processing time in Seconds')

def process_abc():
  while True:
    #Some operations
    c1.set(some_value)
    c2.set(some_value)
    time.sleep(10)

def process_xyz():
   while True:
     #Some operations
     c1.set(some_value)
     c2.set(some_value)
     time.sleep(10)

def main():
  prom.start_http_server(8080)

  Process(target=process_abc).start()
  Process(target=process_xyz).start()

if __name__ == "__main__":
  main()

I was able to see metrics name at endpoint but count is always zero means it's never updated by sub-process.

Answer 1

The prometheus_client library documentation addresses this case:

Prometheus client libraries presume a threaded model, where metrics are shared across workers. This doesn't work so well for languages such as Python where it's common to have processes rather than threads to handle large workloads.

I won't copy here the explanation (which is geared toward gunicorn) use case but basically, you need to:

• define an env variable with directory to use: since you are using Process yourself, you can set it in the code

os.environ["PROMETHEUS_MULTIPROC_DIR"] = "/path/to/writeable/tmp/"

• each process must have its own collector, register it at start and unregister it at exit:

from prometheus_client import multiprocess

def called_from_process():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(CollectorRegistry())

def process_exit(process):
    if process.pid is not None:
        multiprocess.mark_process_dead(process.pid)

p = Process(target=f)
# f calls called_from_process
p.start()
p.join()
process_exit(process)

See the full documentation about how to handle gauge and any other quirk.

I expect that PROMETHEUS_MULTIPROC_DIR should be cleaned at startup of your application to handle odd case where the application previous run was not able to do so.