how to use spark streaming + kafka with streamingListener

我的梦境 提交于 2019-12-12 03:24:23

问题


i have one situation here.

I want that my aplication connect one time to kafka, read the offset, make an action and then stop the application.

I was reading about StreamingListener for detect when the first iteration occurs.

i don´t know how use StreamingListener to stop mi application. can you help me?

I am using spark 1.4

Example code bellow:

val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2)

lines.foreachRDD( rdd => {
  rdd.saveAsTextFile("......")
})


sys.ShutdownHookThread {
  println("Gracefully stopping Spark Streaming Application")
  ssc.stop(true, true)
  println("Application stopped")
}

ssc.start()
ssc.awaitTermination()

thanks!

来源:https://stackoverflow.com/questions/36778566/how-to-use-spark-streaming-kafka-with-streaminglistener

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!