Collect metrics from dockers in Kubernetes

Question

I work with Docker and Kubernetes. I would like to collect application specific metrics from each Docker. There are various applications, each running in one or more Dockers. I would like to collect the metrics in JSON format in order to perform further processing on each type of metrics.

I am trying to understand what is the best practice, if any and what tools can I use to achieve my goal.

Currently I am looking into several options, none looks too good:

1. Connecting to kubectl, getting a list of pods, performing a command (exec) at each pod to cause the application to print/send JSON with metrics. I don't like this option as it means that I need to be aware to which pods exist and access each, while the whole point of having Kubernetes is to avoid dealing with this issue.

2. I am looking for Kubernetes API HTTP GET request that will allow me to pull a specific file. The closest I found is GET /api/v1/namespaces/{namespace}/pods/{name}/log and it seems it is not quite what I need. And again, it forces me to mention each pop by name.

3. I consider to use ExecAction in Probe to send JSON with metrics periodically. It is a hack (as this is not the purpose of Probe), but it removes the need to handle each specific pod

4. I can't use Prometheus for reasons that are out of my control but I wonder how Prometheus collects metric. Maybe I can use similar approach?

Any possible solution will be appreciated.

Answer 1

From an architectural point of view you have 2 options here:

1) pull model : your application exposes metrics data through a mechanisms (for instance using the HTTP protocol on a different port) and an external tool scrapes your pods at a timed interval (getting pod addresses from the k8s API); this is the model used by prometheus for instance.

2) push model : your application actively pushes metrics to an external server, tipically a time series database such as influxdb, when it is most relevant to it.

In my opinion, option 2 is the easiest to implement, because:

• you don't need to deal with k8s APIs in order to discover pods addresses;
• you don't need to create a local storage layer to store your metrics (because you push them one by one as they occour);

But there is a drawback: you need to be careful how you implement this, it could cause your API to become slower and you might need to deal with asynchronous calls to your metrics server.

This is obviously a very generic answer, but I hope it could point you in the right direction.

Answer 2

Pity you can not use Prometheus, but it's a good lead for what can be done in this scope. What Prom does is as follows :

1: it assumes that metrics you want to scrape (collect) are exposed with some http endpoint that Prometheus can access.

2: it connects to kubernetes api to perform a discovery of endpoints to scrape metrics from (there is a config for it, but generaly it means it has to be able to connect to the API and list services/deployments/pods and analyze their annotations (as they have info about metrics endpoints) to compose a list of places to scrape data from

3: periodicaly (15s, 60s etc.) it connects to the endpoints and collects the exposed metrics.

That's it. Rest is storage/postprocessing. The kube related part might be a significant amount of work to do though, so it would be way better to go with something that already exists.

Sidenote: while this is generaly a pull based model, there are cases where pull is not possible (vide short lived scripts like php), that is where Prometheus pushgateway comes into play to allow pushing metrics to a place where Prometheus will pull from.