Spark Metrics: how to access executor and worker data?

Question

Note: I am using Spark on YARN

I have been trying out the Metric System implemented in Spark. I enabled the ConsoleSink and the CsvSink, and enabled JvmSource f

Answer 1

Since you have not given the command you tried, I am assuming that you are not passing metrics.properties. To pass the metrics.propertis the command should be

spark-submit <other parameters> --files metrics.properties 
--conf spark.metrics.conf=metrics.properties

Note metrics.properties has to be specified in --files & --conf, the --files will transfer the metrics.properties file to the executors. Since you can see the output on driver and not on executors I think you are missing the --files option.