Spark streaming using MQTTutils to subscribe topic from activemq with authentication

问题

It seems MQTTUtils Only provide three methods, def createStream(jssc: JavaStreamingContext, brokerUrl: String, topic: String, storageLevel: StorageLevel): JavaDStream[String]

Create an input stream that receives messages pushed by a MQTT publisher. def createStream(jssc: JavaStreamingContext, brokerUrl: String, topic: String): JavaDStream[String]

Create an input stream that receives messages pushed by a MQTT publisher. def createStream(ssc: StreamingContext, brokerUrl: String, topic: String, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): DStream[String]

Create an input stream that receives messages pushed by a MQTT publisher.

But How can I provide username and password if the broker enabled authentication?

回答1:

You could try including the username and password in the url:

mqtt://username:password@host:port

回答2:

Please find this MQTT Scala Word Count Example.

Particular for your case run the publisher as

bin/run-example org.apache.spark.examples.streaming.MQTTPublisher mqtt://username:password@host:port foo

And Subscriber as

bin/run-example org.apache.spark.examples.streaming.MQTTWordCount mqtt://username:password@host:port foo

Before doing this ensure that you are started ActiveMQ broker.

example code

import org.apache.activemq.broker.{TransportConnector, BrokerService}
.
.
.
.
def startActiveMQMQTTBroker() {
    broker = new BrokerService()
    broker.setDataDirectoryFile(Utils.createTempDir())
    connector = new TransportConnector()
    connector.setName("mqtt")
    connector.setUri(new URI("mqtt:" + brokerUri))
    broker.addConnector(connector)
    broker.start()
}

pom file

<dependency>
  <groupId>org.apache.activemq</groupId>
  <artifactId>activemq-core</artifactId>
  <version>5.7.0</version>
</dependency>

回答3:

You can trying using the customized spark-streaming-mqtt-connector library available here - https://github.com/sathipal/spark-streaming-mqtt-with-security_2.10-1.3.0.

This library adds the following on top of the original library,

Added TLS v1.2 security such that the communication is always secured.
Stored topic along with the payload in the RDD.

So, use the following method to create the stream,

val lines = MQTTUtils.createStream(ssc, // Spark Streaming Context
            "ssl://URL",                // Broker URL
            "<topic>",                 // MQTT topic
            "MQTT client-ID",          // Unique ID of the application
            "Username", 
            "passowrd")

There are overloaded constructors that allows you to pass the RDD storage level as well. Hope this helps.

来源：https://stackoverflow.com/questions/27777998/spark-streaming-using-mqttutils-to-subscribe-topic-from-activemq-with-authentica

标签

authentication

mqtt

spark-streaming