Flink metrics as csv?

Question

I'm trying to test different metrics on some flink jobs. However, I found it pretty cumbersome to set up a metrics reporter. Currently I'm using Graphite and for each job I need to manually select all the metrics I want, put them into different graphs to get an overview whether this test is meaningful and if so, I have to export all metrics individually into a csv file and finally merge it again in another diagram (custom diagram is mandatory unfortunately).

Then I redeploy the job with different settings or parallism, which changes the job id, which means I need to put all graphs together again, export those metrics, and so on...

Is there a more comfortable way to get metrics of flink jobs as csv file? Would be nice to get the metrics of certain operators (those ids stay the same) somehow as csv, independent of the job id or taskmanager id.

Answer 1

With the influxdb reporter it's pretty easy to setup scope formats that make it the metric naming really clean, and then you can do nice regular queries on the database to dump out groups of metrics. See https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter .