Spark Streaming MQTT | 易学教程

问题

I've been using spark to stream data from kafka and it's pretty easy.

I thought using the MQTT utils would also be easy, but it is not for some reason.

I'm trying to execute the following piece of code.

  val sparkConf = new SparkConf(true).setAppName("amqStream").setMaster("local")
  val ssc = new StreamingContext(sparkConf, Seconds(10))

  val actorSystem = ActorSystem()
  implicit val kafkaProducerActor = actorSystem.actorOf(Props[KafkaProducerActor])

  MQTTUtils.createStream(ssc, "tcp://localhost:1883", "AkkaTest")
    .foreachRDD { rdd =>
      println("got rdd: " + rdd.toString())
      rdd.foreach { msg =>
        println("got msg: " + msg)
      }
    }

  ssc.start()
  ssc.awaitTermination()

The weird thing is that spark logs the msg I sent in the console, but not my println.

It logs something like this:

19:38:18.803 [RecurringTimer - BlockGenerator] DEBUG o.a.s.s.receiver.BlockGenerator - Last element in input-0-1435790298600 is SOME MESSAGE

回答1:

foreach is a distributed action, so your println may be executing on the workers. If you want to see some of the messages printed out locally, you could use the built in print function on the DStream or instead of your foreachRDD collect (or take) some of the elements back to the driver and print them there. Hope that helps and best of luck with Spark Streaming :)

回答2:

If you wish to just print incoming messages, try something like this instead of the for_each (translating from a working Python version, so do check for Scala typos):

val mqttStream = MQTTUtils.createStream(ssc, "tcp://localhost:1883", "AkkaTest")
mqttStream.print()

来源：https://stackoverflow.com/questions/31172858/spark-streaming-mqtt

标签

scala

apache-spark

mqtt

spark-streaming