Kryo serialization library: is it used in production?

拟墨画扇 提交于 2019-11-27 00:03:37

问题


Kryo is a very new and interesting Java serialization library, and one of the fastest in the thrift-protobuf benchmark. If you've used Kryo, has it already reached enough maturity to try it out in production code?

Update (10/27/2010): We're using Kryo, though not yet in production. See my answer below for details.

Update (3/9/2011): Updating to the latest Jackson and Kryo libraries shows that Jackson's binary Smile serialization is pretty competitive.


回答1:


There is a bug report and a discussion thread. The DateSerializer that comes with Kryo is slightly more efficient size-wise than the SimpleSerializer implementation posted on SO because it uses LongSerializer optimized for positive values.

Edit: I forgot to answer the original question. I believe Kryo is used in at least a few production systems. There is mention of it in this article, Jive SBS cache redesign: Part 3. In the Destroy All Humans project, Kryo is used to communicate with an Android phone that serves as a robot brain (video here).

Not a direct answer, but you might browse the Kryo source and/or javadocs. Check out the read* and write* methods on the Kryo class, then look at the Serializer class. This is really the core of the library.




回答2:


I'll try to answer my own question (Kyro is still very new!).

We have a set of about 120 different web services implemented using the Restlet framework. These are consumed by web service clients generally built on top of a Restlet-based client library. The representations sent back and forth between server and client include XML (using the XStream serialization library), JSON (Using Jackson), XHTML, Java Object Serialization, and as of yesterday, Kryo. So we're in a position to do some solid side-by-side comparisons.

Kryo 1.0.1 seems reasonably stable. Once I actually read up on how to use the API, the only real problem I found was that the default java.util.Date serializer seemed to warp dates a few months into the past. I just had to provide my own override:

kryo.register(Date.class, 
  new SimpleSerializer<Date>() {
   @Override public void write (ByteBuffer b, Date d) { b.putLong(d.getTime()); }
   @Override public Date read (ByteBuffer b) { return new Date(b.getLong()); }
  });

But that was the only possible issue I've found so far. We have a set of JavaBeans that have String, Float, Integer, Long, Date, Boolean and List fields.

Here are some rough benchmarks. First, I did 100,000 serializations and deserializations of an object hierarchy that describes one TV program (ie, made 100,000 deep copies of it). The speeds were:

XStream XML:                 360/sec
Java Object Serialization: 1,570/sec
Jackson JSON:              5,000/sec
Kryo:                      8,100/sec

Next, I also serialized a catalog of 2,000 TV program descriptions and counted bytes:

XStream XML:         6,837,851 bytes
Jackson JSON:        3,656,654 bytes
Kryo:                1,124,048 bytes

I also found that registering serializers was very important:

kryo.register(List.class);
kryo.register(ArrayList.class);
// ...
kryo.register(Program.class);
kryo.register(Catalog.class);
// ...

If I didn't do that, the serializations were almost double the size, and the speed was maybe 40% slower.

We also ran complete end-to-end tests of several web services using each of these four serialization methods, and they also showed that Kryo was running faster than the others.

So in summary, Kryo seems reasonably robust. I'm going to keep support for it in our code base and as we gain experience with it I hope to use it in more places. Kudos to the Kryo team!

Update (3/9/2011): I finally got around to @StaxMan's suggestion to try Jackson 1.6's binary "Smile" serializer. Using Jackson 1.6 and Kryo 1.04, I did 100,000 deep copies (serialization/deserialiations) of a somewhat different TV program object hierarchy:

XStream XML:     429/sec    5,189 bytes
Jackson JSON:  4,474/sec    2,657 bytes
Kryo:          4,539/sec    1,066 bytes  
Jackson Smile: 5,040/sec    1,689 bytes

This test didn't mesh with a macro-level test, where I tried different serializers in a REST web service that delivers many of these objects. There the overall system throughput supports @StaxMan's intuition about performance:

Jackson JSON:     92 requests/sec
Jackson Smile     97 requests/sec
Kryo:            108 requests/sec



回答3:


Kryo is part of Yahoo's S4 (Simple Scalable Streaming System) project. S4 isn't production yet as far as I know.




回答4:


With the help of Jim Ferrans responses and comments above I found a more detailed explanation about Date Serialization Issue with Kryo on this page: http://groups.google.com/group/kryo-users/browse_thread/thread/91969c6f48a45bdf/ and also a how to use DateSerializer() of Kryo:

kryo.register(Date.class, new DateSerializer());

I hope this could help others.




回答5:


The latest version of Kryo has a few race conditions in some extreme cases, running on a simulator interface to ns-3 from Java. Might ask the developer to commit some of my changes back if they are problem free.




回答6:


Apache Storm uses it for serialization before passing messages from one task to another.

So yes it must be quite stable since Storm is used by several huge companies, i.e., Twitter and Spotify.




回答7:


Kryo 2.x is also used by Mule ESB, and so widely used in production.




回答8:


2017 update:

Kryo is used by Flink. So practically anything that is using Flink framework is relying on Kryo. Reference: https://ci.apache.org/projects/flink/flink-docs-release-0.8/programming_guide.html#specifying-keys




回答9:


The Kryo site has section on projects in production using Kryo



来源:https://stackoverflow.com/questions/2725233/kryo-serialization-library-is-it-used-in-production

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!