When to use Kryo serialization in Spark?

Question

I am already compressing RDDs using conf.set(\"spark.rdd.compress\",\"true\") and persist(MEMORY_AND_DISK_SER). Will using Kryo serialization make the

Answer 1

Kryo serialization is a more optimized serialization technique so you can use it to serialize any class which is used in an RDD or Dataframe closure. For some specific information use of Kryo serialization, see below:

1. Use when serializing third party non-serialize classes inside an RDD or dataframe closure
2. You want to use efficient serialization technique
3. If you ever got a serialization error because of some class, you can register that class with the Kryo serializer