kryo

问题 I am trying to use Spark with Kryo Serializer to store some data with less memory cost. And now I come across a trouble, I cannot save a DataFram e(whose type is Dataset[Row]) in memory with Kryo serializer. I thought all I need to do is to add org.apache.spark.sql.Row to classesToRegister , but error still occurs: spark-shell --conf spark.kryo.classesToRegister=org.apache.spark.sql.Row --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrationRequired

问题 I am trying to use Spark with Kryo Serializer to store some data with less memory cost. And now I come across a trouble, I cannot save a DataFram e(whose type is Dataset[Row]) in memory with Kryo serializer. I thought all I need to do is to add org.apache.spark.sql.Row to classesToRegister , but error still occurs: spark-shell --conf spark.kryo.classesToRegister=org.apache.spark.sql.Row --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrationRequired

问题 I made a class Person and registered it but on runtime, it shows class not registered.Why is it showing so? Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 0, not attempting to retry it. Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException: Class is not registered: KyroExample$Person[] Note: To register this class use: kryo.register(KyroExample$Person[].class); Here is the sample code :

问题 I am trying to register a class with array (Spark Java with Kryo activated), log shows a clear message: Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[] I have written several combinations, but these do not work: kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]")); // ERROR kryo.register(Class.forName("org.apache.spark.sql.execution.datasources

问题 I am trying to register a class with array (Spark Java with Kryo activated), log shows a clear message: Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[] I have written several combinations, but these do not work: kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]")); // ERROR kryo.register(Class.forName("org.apache.spark.sql.execution.datasources

问题 I am running a program on Apache Flink. I got this error: Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: Serializer consumed more bytes than the record had. This indicates broken serialization. If you are using custom serialization types (Value or Writable), check their serialization methods. If you are using a Kryo-serialized type, check the corresponding Kryo serializer. How can I check the serialization method of an object in Scala/Java?

问题 While running a Gradle build, tests are failing. PS: 1. Gradle is using the correct JDK (1.6) to build. 2. I tried this with JDK 1.7, same error comes there as well. 3. I don't see this error when I build it locally (with JDK 1.6) on a linux/windows 4. machine but one of the machine is giving me this error. My ?s 1. What can be done to fix the com.esotericsoftware.kryo.KryoException: Buffer overflow error. 2. Why Gradle process failed, even when test section in build.gradle says: test {

问题 I try to use the spark shell to connect to an Accumulo Table I load spark and the libraries I need like this: $ bin/spark-shell --jars /data/bigdata/installs/accumulo-1.7.2/lib/accumulo-fate.jar:/data/bigdata/installs/accumulo-1.7.2/lib/accumulo-core.jar:/data/bigdata/installs/accumulo-1.7.2/lib/accumulo-trace.jar:/data/bigdata/installs/accumulo-1.7.2/lib/htrace-core.jar:/data/bigdata/installs/accumulo-1.7.2/lib/libthrift.jar To the shell, I paste import org.apache.hadoop.mapred.JobConf

问题 I am hitting a bug when using java protocol buffer classes as the object model for RDDs in Spark jobs, For my application, my ,proto file has properties that are repeated string. For example message OntologyHumanName { repeated string family = 1; } From this, the 2.5.0 protoc compiler generates Java code like private com.google.protobuf.LazyStringList family_ = com.google.protobuf.LazyStringArrayList.EMPTY; If I run a Scala Spark job that uses the Kryo serializer I get the following error

问题 I am seeing an error when running my spark job relating to Serialization of a protobuf field when transforming an RDD. com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException Serialization trace: otherAuthors_ (com.thomsonreuters.kraken.medusa.dbor.proto.Book$DBBooks) The error seems to be created at this point: val booksPerTier: Iterable[(TimeTier, RDD[DBBooks])] = allTiers.map { tier => (tier, books.filter(b => isInTier(endOfInterval, tier, b) && !isBookPublished(o)