如何创建一个源以将指标从 Spark 导出到另一个接收器（Prometheus）？

Question

I am trying to create a source for metrics from my spark application written in Scala to export data to another system, preferable to Prometheus. According to this site from Data bricks I need to create a source that extends the Source trait. However, the Source trait is private[spark] trait Source and my source cannot visualize it. When I create this class I get the error Symbol Source is inaccessible from this place .

package org.sense.spark.util

import org.apache.spark.metrics.source.Source
import com.codahale.metrics.{Counter, Histogram, MetricRegistry}

class MetricSource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

  val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))
  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

How can I create my source to export data to Prometheus? I would like to export monitored values from a UDF inside the combineByKey transformation. The values would be latency to aggregate and throughput IN/OUT of this transformation.

This is my build.sbt file in case it is necessary to check the libraries that I am using.

name := "explore-spark"

version := "0.2"

scalaVersion := "2.12.3"

val sparkVersion = "3.0.0"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.twitter" %% "algebird-core" % "0.13.7",
  "joda-time" % "joda-time" % "2.5",
  "org.fusesource.mqtt-client" % "mqtt-client" % "1.16"
)

mainClass in(Compile, packageBin) := Some("org.sense.spark.app.App")
mainClass in assembly := Some("org.sense.spark.app.App")

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyJarName in assembly := s"${name.value}_${scalaBinaryVersion.value}-fat_${version.value}.jar"

Answer 1

You will need to put your class which extends Source in the same package as source

package org.apache.spark.metrics.source

import com.codahale.metrics.{Counter, Histogram, MetricRegistry}

class MetricsSource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

    val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))

  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

Answer 2

if in case you need to use prometheus sink instead of console sink, you can use third party library which was written for spark prometheus sink. this works via pushgateway- https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md