Kryo serialization library: is it used in production?

Question

Kryo is a very new and interesting Java serialization library, and one of the fastest in the thrift-protobuf benchmark. If you've used Kryo, has it already reached enough maturity to try it out in production code?

Update (10/27/2010): We're using Kryo, though not yet in production. See my answer below for details.

Update (3/9/2011): Updating to the latest Jackson and Kryo libraries shows that Jackson's binary Smile serialization is pretty competitive.

Answer 1

There is a bug report and a discussion thread. The DateSerializer that comes with Kryo is slightly more efficient size-wise than the SimpleSerializer implementation posted on SO because it uses LongSerializer optimized for positive values.

Edit: I forgot to answer the original question. I believe Kryo is used in at least a few production systems. There is mention of it in this article, Jive SBS cache redesign: Part 3. In the Destroy All Humans project, Kryo is used to communicate with an Android phone that serves as a robot brain (video here).

Not a direct answer, but you might browse the Kryo source and/or javadocs. Check out the read* and write* methods on the Kryo class, then look at the Serializer class. This is really the core of the library.

Answer 2

I'll try to answer my own question (Kyro is still very new!).

We have a set of about 120 different web services implemented using the Restlet framework. These are consumed by web service clients generally built on top of a Restlet-based client library. The representations sent back and forth between server and client include XML (using the XStream serialization library), JSON (Using Jackson), XHTML, Java Object Serialization, and as of yesterday, Kryo. So we're in a position to do some solid side-by-side comparisons.

Kryo 1.0.1 seems reasonably stable. Once I actually read up on how to use the API, the only real problem I found was that the default java.util.Date serializer seemed to warp dates a few months into the past. I just had to provide my own override:

kryo.register(Date.class, 
  new SimpleSerializer<Date>() {
   @Override public void write (ByteBuffer b, Date d) { b.putLong(d.getTime()); }
   @Override public Date read (ByteBuffer b) { return new Date(b.getLong()); }
  });

But that was the only possible issue I've found so far. We have a set of JavaBeans that have String, Float, Integer, Long, Date, Boolean and List fields.

Here are some rough benchmarks. First, I did 100,000 serializations and deserializations of an object hierarchy that describes one TV program (ie, made 100,000 deep copies of it). The speeds were:

XStream XML:                 360/sec
Java Object Serialization: 1,570/sec
Jackson JSON:              5,000/sec
Kryo:                      8,100/sec

Next, I also serialized a catalog of 2,000 TV program descriptions and counted bytes:

XStream XML:         6,837,851 bytes
Jackson JSON:        3,656,654 bytes
Kryo:                1,124,048 bytes

I also found that registering serializers was very important:

kryo.register(List.class);
kryo.register(ArrayList.class);
// ...
kryo.register(Program.class);
kryo.register(Catalog.class);
// ...

If I didn't do that, the serializations were almost double the size, and the speed was maybe 40% slower.

We also ran complete end-to-end tests of several web services using each of these four serialization methods, and they also showed that Kryo was running faster than the others.

So in summary, Kryo seems reasonably robust. I'm going to keep support for it in our code base and as we gain experience with it I hope to use it in more places. Kudos to the Kryo team!

Update (3/9/2011): I finally got around to @StaxMan's suggestion to try Jackson 1.6's binary "Smile" serializer. Using Jackson 1.6 and Kryo 1.04, I did 100,000 deep copies (serialization/deserialiations) of a somewhat different TV program object hierarchy:

XStream XML:     429/sec    5,189 bytes
Jackson JSON:  4,474/sec    2,657 bytes
Kryo:          4,539/sec    1,066 bytes  
Jackson Smile: 5,040/sec    1,689 bytes

This test didn't mesh with a macro-level test, where I tried different serializers in a REST web service that delivers many of these objects. There the overall system throughput supports @StaxMan's intuition about performance:

Jackson JSON:     92 requests/sec
Jackson Smile     97 requests/sec
Kryo:            108 requests/sec

Answer 3

Kryo is part of Yahoo's S4 (Simple Scalable Streaming System) project. S4 isn't production yet as far as I know.

Answer 4

With the help of Jim Ferrans responses and comments above I found a more detailed explanation about Date Serialization Issue with Kryo on this page: http://groups.google.com/group/kryo-users/browse_thread/thread/91969c6f48a45bdf/ and also a how to use DateSerializer() of Kryo:

kryo.register(Date.class, new DateSerializer());

I hope this could help others.

Answer 5

The latest version of Kryo has a few race conditions in some extreme cases, running on a simulator interface to ns-3 from Java. Might ask the developer to commit some of my changes back if they are problem free.

Answer 6

Apache Storm uses it for serialization before passing messages from one task to another.

So yes it must be quite stable since Storm is used by several huge companies, i.e., Twitter and Spotify.

Answer 7

The Kryo site has section on projects in production using Kryo

Answer 8

Kryo 2.x is also used by Mule ESB, and so widely used in production.

Answer 9

2017 update:

Kryo is used by Flink. So practically anything that is using Flink framework is relying on Kryo. Reference: https://ci.apache.org/projects/flink/flink-docs-release-0.8/programming_guide.html#specifying-keys