Error reading Kafka SSL client truststore file from Spark streaming

问题

I have a Spark streaming application reading from Kafka. I am running it from EMR. Recently I implemented Kafka SSL. I am creating the Kafka client as shown below. I am getting a strange error running the application when it tries to read the truststore file. Error is:

- Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.FileNotFoundException: /tmp/kafka.client.truststore.jks (No such file or directory)

What is causing this issue?

DataStreamReader df = session.readStream()
    .format("kafka")
    .option("kafka.bootstrap.servers",kafka_server)
    .option("subscribe", kafka_topic)
    .option("failOnDataLoss", "false")
    .option("group.id", kafka_group)
    .option("kafka.security.protocol","SSL")
    .option("kafka.ssl.truststore.location","/tmp/kafka.client.truststore.jks")
    .option("kafka.ssl.truststore.password","clientpass");

回答1:

That file must exist on the executor nodes of the cluster, not just on the machine you submit the code from

Otherwise, you must pass the --files option to spark-submit to pass them from your machine to the driver

Also note: group.id isn't allowed in structured streaming

来源：https://stackoverflow.com/questions/59274128/error-reading-kafka-ssl-client-truststore-file-from-spark-streaming