问题
How can I register MapWithStateRDDRecord
in kryo?
When I'm trying to do.
`sparkConfiguration.registerKryoClasses(Array(classOf[org.apache.spark.streaming.rdd.MapWithStateRDD))`
I get an error
class MapWithStateRDDRecord in package rdd cannot be accessed in package org.apache.spark.streaming.rdd
[error] classOf[org.apache.spark.streaming.rdd.MapWithStateRDDRecord]
I'd like to make sure that all serialization is done via kryo thus I set SparkConf().set("spark.kryo.registrationRequired", "true")
. With this setting enabled I get exceptions during runtime. java.lang.IllegalArgumentException (Class is not registered: org.apache.spark.streaming.rdd.MapWithStateRDDRecord)
来源:https://stackoverflow.com/questions/42555183/mapwithstaterddrecord-with-kryo