How to implement custom job listener/tracker in Spark?

Question

I have a class like below, and when i run this through command line i want to see progress status. some thing like,

10% completed... 
30% completed... 
100%         


        

Answer 1

If you are using scala-spark this code will help you to adding spark listener.

Create your SparkContext

val sc=new SparkContext(sparkConf) 

Now you can add your spark listener in spark context

sc.addSparkListener(new SparkListener() {
  override def onApplicationStart(applicationStart: SparkListenerApplicationStart) {
    println("Spark ApplicationStart: " + applicationStart.appName);
  }

  override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
    println("Spark ApplicationEnd: " + applicationEnd.time);
  }

});

Here is the list of Interface for listening to events from the Spark schedule.

Answer 2

First thing is if you want track progress then you can consider spark.ui.showConsoleProgress pls check @Yijie Shens answer(Spark output: log-style vs progress-style) for this..

I think no need to implement Spark listener for such thing. Unless you are very specific.

---

Question : How to implement custom job listener/tracker in Spark?

You can Use SparkListener and intercept SparkListener events.

Classic example of this implementation with in Spark Framework it self is HeartBeatReceiver.

Example : HeartBeatReceiver.scala

/**
 * Lives in the driver to receive heartbeats from executors..
 */
private[spark] class HeartbeatReceiver(sc: SparkContext, clock: Clock)
  extends SparkListener with ThreadSafeRpcEndpoint with Logging {

  def this(sc: SparkContext) {
    this(sc, new SystemClock)
  }

  sc.addSparkListener(this) ...

Below are list of Listener events available. out of which application/job events should be useful for you

• SparkListenerApplicationStart

• SparkListenerJobStart

• SparkListenerStageSubmitted

• SparkListenerTaskStart

• SparkListenerTaskGettingResult

• SparkListenerTaskEnd

• SparkListenerStageCompleted

• SparkListenerJobEnd

• SparkListenerApplicationEnd

• SparkListenerEnvironmentUpdate

• SparkListenerBlockManagerAdded

• SparkListenerBlockManagerRemoved

• SparkListenerBlockUpdated

• SparkListenerUnpersistRDD

• SparkListenerExecutorAdded

• SparkListenerExecutorRemoved

Answer 3

You should implement SparkListener. Just override whatever events you are interested in (job/stage/task start/end events), then call sc.addSparkListener(myListener).

It does not give you a straight-up percentage-based progress tracker, but at least you can track that progress is being made and its rough rate. The difficulty comes from how unpredictable the number of Spark stages can be, and also how the running times of each stage can be vastly different. The progress within a stage should be more predictable.