Flink statsD 集成

Question

We are using Flink 1.9.1 on EMR 5.29 with multi-master mode enabled. We added the below configuration to the flink-conf.yaml file in conf folder and added the jar flink-metrics-statsd-1.9.1.jar to the flink/lib folder.

metrics.scope.jm: <host>.jobmanager
metrics.scope.jm.job: <host>.jobmanager.<job_name>
metrics.scope.tm: <host>.taskmanager.<tm_id>
metrics.scope.tm.job: <host>.taskmanager.<tm_id>.<job_name>
metrics.scope.task: <host>.taskmanager.<tm_id>.<job_name>.<task_name>.<subtask_index>
metrics.scope.operator: <host>.taskmanager.<tm_id>.<job_name>.<operator_name>.<subtask_index>

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporter
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8130
metrics.reporter.stsd.interval: 60 SECONDS

When we listen to the statsD port, we don't get any of the metrics. Is there anything that we are missing out? Is there a way to check if some parameter is not getting passed?

As suggested in one of the other answers, we tried giving the IP address of the master also. Does not seem to work.

Answer 1

With Flink 1.9, you need to copy the jar file for the reporter into the lib directory of every machine in the cluster. So copy flink-metrics-statsd-1.9.1.jar from opt into lib .

If that doesn't fix it, check the logs -- there should be clues there.

Also, statsd normally uses port 8125. Are you sure it's listening on 8130?

Answer 2

Found the issue: This should have been used: metrics.reporter.stsd.class: org.apache.flink.metrics.statsd.StatsDReporter Instead of: metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporter