I can't do transformation on a RDD[ConsumerRecord]

Question

I have a serialization problem with a ConsumerRecord recover from a kafka topic in a DStream. To illustrate my problem, I create the following exemple. I create a ConsumerRecord

Answer 1

org.apache.kafka.clients.consumer.ConsumerRecord is not serializable, So there are some way to solve this kind of problems

1. Serializable the class using kryo serialization
2. Declare the instance only within the lambda function passed in map.
3. Make the NotSerializable object as a static and create it once per machine.

4. Call rdd.forEachPartition and create the NotSerializable object in there like this:

   rdd.forEachPartition(iter -> { NotSerializable notSerializable = new NotSerializable();

   // ...Now process iter });

For more details you can check this link https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/troubleshooting/javaionotserializableexception.html