kryo

Twitter-chill looks like a good solution to the problem of how to serialize efficiently in Scala without excessive boilerplate. However, I don't see any evidence of how they handle case classes. Does this just work automatically or does something need to be done (e.g. creating a zero-arg constructor)? I have some experience with the WireFormat serialization mechanism built into Scoobi, which is a Scala Hadoop wrapper similar to Scalding. They have serializers for case classes up to 22 arguments that use the apply and unapply methods and do type matching on the arguments to these functions to

I am already compressing RDDs using conf.set("spark.rdd.compress","true") and persist(MEMORY_AND_DISK_SER) . Will using Kryo serialization make the program even more efficient, or is it not useful in this case? I know that Kryo is for sending the data between the nodes in a more efficient way. But if the communicated data is already compressed, is it even needed? Tim Both of the RDD states you described (compressed and persisted) use serialization. When you persist an RDD, you are serializing it and saving it to disk (in your case, compressing the serialized output as well). You are right that

Please help to understand how Kryo serializer allocates memory for its buffer. My Spark app fails on a collect step when it tries to collect about 122Mb of data to a driver from workers. com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 57197 at com.esotericsoftware.kryo.io.Output.require(Output.java:138) at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:220) at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:206) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:29) at

I use Kryo to write Objects into byte arrays. It works fine. But when the byte arrays are converted into the Objects, it throws, com.esotericsoftware.kryo.KryoException: Buffer underflow. exception. This is my deserialization: Kryo k=new Kryo(); Input input=new Input(byteArrayOfObject); Object o=k.readObject(input,ObjectClass.class); Furthermore, always the object type cannot be defined in my application. At the final process, the class conversion happens. Therefore, How can I solve above deserialization error Is there a way to create Object without giving the class into readObject(...

Problem description We have a Hadoop cluster on which we store data which is serialized to bytes using Kryo (a serialization framework). The Kryo version which we used to do this has been forked from the official release 2.21 to apply our own patches to issues we have experienced using Kryo. The current Kryo version 2.22 also fixes these issues, but with different solutions. As a result, we cannot just change the Kryo version we use, because this would mean that we would no longer be able to read the data which is already stored on our Hadoop cluster. To address this problem, we want to run a