Spark: configuration file 'metrics.properties'

Question

Spark's configuration file (useful to retrieve metrics), namely /conf/metrics.properties , states what follows:

Within an instance, a "source" specifies a particular set of grouped metrics. there are two kinds of sources:

1. Spark internal sources, like MasterSource, WorkerSource, etc, which will collect a Spark component's internal state. Each instance is paired with a Spark source that is added automatically.

2. Common sources, like JvmSource, which will collect low level state. These can be added through configuration options and are then loaded using reflection.

Some examples follow, as

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Yet, there is no example explaining how to get metrics from other sources as MasterSource or WorkerSource , and just replacing JvmSource is not doing the trick.

Suggestions?

Answer 1

MasterSource or WorkerSource --which are both internal sources-- are automatically traced by metrics, so there is no need to specify this explicitly.

For instance,

master.sink.cvs.class=org.apache.spark.metrics.sink.CsvSink

will output master instance metrics to /tmp directory each 10 seconds.