[英]Spark Metrics: how to access executor and worker data?
Note: I am using Spark on YARN 注意:我在YARN上使用Spark
I have been trying out the Metric System implemented in Spark. 我一直在尝试在Spark中实现的度量系统 。 I enabled the ConsoleSink and the CsvSink, and enabled JvmSource for all four instances (driver, master, executor, worker).
我启用了ConsoleSink和CsvSink,并为所有四个实例(驱动程序,主机,执行程序,工作程序)启用了JvmSource。 However I only have driver outputs, and no worker/executor/master data in the console and csv target directory.
但是我只有驱动程序输出,并且在控制台和csv目标目录中没有worker / executor / master数据。
After having read this question , I wonder if I do have to ship something to executors when submitting a job. 阅读完此问题后 ,我想知道提交工作时是否必须将某些物品运送给执行者。
My submit command: ./bin/spark-submit --class org.apache.spark.examples.SparkPi lib/spark-examples-1.5.0-hadoop2.6.0.jar 10
我的提交命令:
./bin/spark-submit --class org.apache.spark.examples.SparkPi lib/spark-examples-1.5.0-hadoop2.6.0.jar 10
submit ./bin/spark-submit --class org.apache.spark.examples.SparkPi lib/spark-examples-1.5.0-hadoop2.6.0.jar 10
Bellow is my metric.properties file: 波纹管是我的metric.properties文件:
# Enable JmxSink for all instances by class name
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
# Enable ConsoleSink for all instances by class name
*.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink
# Polling period for ConsoleSink
*.sink.console.period=10
*.sink.console.unit=seconds
#######################################
# worker instance overlap polling period
worker.sink.console.period=5
worker.sink.console.unit=seconds
#######################################
# Master instance overlap polling period
master.sink.console.period=15
master.sink.console.unit=seconds
# Enable CsvSink for all instances
*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
#driver.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
# Polling period for CsvSink
*.sink.csv.period=10
*.sink.csv.unit=seconds
# Polling directory for CsvSink
*.sink.csv.directory=/opt/spark-1.5.0-bin-hadoop2.6/csvSink/
# Worker instance overlap polling period
worker.sink.csv.period=10
worker.sink.csv.unit=second
# Enable Slf4jSink for all instances by class name
#*.sink.slf4j.class=org.apache.spark.metrics.sink.Slf4jSink
# Polling period for Slf4JSink
#*.sink.slf4j.period=1
#*.sink.slf4j.unit=minutes
# Enable jvm source for instance master, worker, driver and executor
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
And here is a listing of the csv files created by Spark. 这是Spark创建的csv文件的列表。 I am looking forward to access the same data for Spark executors (which are also JVMs).
我期待为Spark执行程序(也是JVM)访问相同的数据。
app-20160812135008-0013.driver.BlockManager.disk.diskSpaceUsed_MB.csv
app-20160812135008-0013.driver.BlockManager.memory.maxMem_MB.csv
app-20160812135008-0013.driver.BlockManager.memory.memUsed_MB.csv
app-20160812135008-0013.driver.BlockManager.memory.remainingMem_MB.csv
app-20160812135008-0013.driver.jvm.heap.committed.csv
app-20160812135008-0013.driver.jvm.heap.init.csv
app-20160812135008-0013.driver.jvm.heap.max.csv
app-20160812135008-0013.driver.jvm.heap.usage.csv
app-20160812135008-0013.driver.jvm.heap.used.csv
app-20160812135008-0013.driver.jvm.non-heap.committed.csv
app-20160812135008-0013.driver.jvm.non-heap.init.csv
app-20160812135008-0013.driver.jvm.non-heap.max.csv
app-20160812135008-0013.driver.jvm.non-heap.usage.csv
app-20160812135008-0013.driver.jvm.non-heap.used.csv
app-20160812135008-0013.driver.jvm.pools.Code-Cache.committed.csv
app-20160812135008-0013.driver.jvm.pools.Code-Cache.init.csv
app-20160812135008-0013.driver.jvm.pools.Code-Cache.max.csv
app-20160812135008-0013.driver.jvm.pools.Code-Cache.usage.csv
app-20160812135008-0013.driver.jvm.pools.Code-Cache.used.csv
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.committed.csv
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.init.csv
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.max.csv
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.usage.csv
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.used.csv
app-20160812135008-0013.driver.jvm.pools.Metaspace.committed.csv
app-20160812135008-0013.driver.jvm.pools.Metaspace.init.csv
app-20160812135008-0013.driver.jvm.pools.Metaspace.max.csv
app-20160812135008-0013.driver.jvm.pools.Metaspace.usage.csv
app-20160812135008-0013.driver.jvm.pools.Metaspace.used.csv
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.committed.csv
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.init.csv
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.max.csv
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.usage.csv
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.used.csv
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.committed.csv
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.init.csv
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.max.csv
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.usage.csv
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.used.csv
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.committed.csv
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.init.csv
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.max.csv
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.usage.csv
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.used.csv
app-20160812135008-0013.driver.jvm.PS-MarkSweep.count.csv
app-20160812135008-0013.driver.jvm.PS-MarkSweep.time.csv
app-20160812135008-0013.driver.jvm.PS-Scavenge.count.csv
app-20160812135008-0013.driver.jvm.PS-Scavenge.time.csv
app-20160812135008-0013.driver.jvm.total.committed.csv
app-20160812135008-0013.driver.jvm.total.init.csv
app-20160812135008-0013.driver.jvm.total.max.csv
app-20160812135008-0013.driver.jvm.total.used.csv
DAGScheduler.job.activeJobs.csv
DAGScheduler.job.allJobs.csv
DAGScheduler.messageProcessingTime.csv
DAGScheduler.stage.failedStages.csv
DAGScheduler.stage.runningStages.csv
DAGScheduler.stage.waitingStages.csv
Since you have not given the command you tried, I am assuming that you are not passing metrics.properties. 由于您没有给出尝试的命令,因此我假设您没有传递metrics.properties。 To pass the metrics.propertis the command should be
要传递metrics.propertis,该命令应为
spark-submit <other parameters> --files metrics.properties
--conf spark.metrics.conf=metrics.properties
Note metrics.properties has to be specified in --files & --conf, the --files will transfer the metrics.properties file to the executors. 注意必须在--files和--conf中指定metrics.properties,这些--files会将metrics.properties文件传输到执行程序。 Since you can see the output on driver and not on executors I think you are missing the --files option.
由于您可以在驱动程序而不是执行程序上看到输出,因此我认为您缺少--files选项。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.