Java client library for reading from Kestrel server Queue from within Storm Spout

问题

I've setup a Kestrel server and able to setup and use queues via the python pykestrel library.

We have a scenario where a python client writes to Kestrel queue(s) and a Storm spout needs to read from the queue(s). I've tried using the storm-kestrel library but running into issues. Googling seems to suggest it doesn't support the memcache port (22133).

I've added the maven bindings as provided here. Didn't use the KestrelThriftSpout spout, using Kestrel.Client. Compilation is fine but I get java.lang.NoClassDefFoundError error at runtime.

STACK TRACE

1098 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn  - Thread Thread[main,5,main] died
java.lang.NoClassDefFoundError: Lnet/lag/kestrel/thrift/Kestrel$Client;
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
    at java.lang.Class.getDeclaredField(Class.java:1946)
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at backtype.storm.utils.Utils.serialize(Utils.java:52)
    at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94)
    at com.rr.storm.ArticleTolopolgy.main(ArticleTolopolgy.java:35)
Caused by: java.lang.ClassNotFoundException: net.lag.kestrel.thrift.Kestrel$Client
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 15 more

Can someone please confirm if storm-kestrel can or cannot be used in this scenario?

If not, is there another kestrel client library that is recommended to be used from within a storm spout for reading data off of kestrel queue(s)?

回答1:

Sorry for the late update.

The java.lang.NoClassDefFoundError was because some JARs were not in classpath. There were several dependencies involved so we resolved the issue by packing all dependencies into a SINGLE JAR. There were several dependency conflicts that needed to be resolved esp. a nasty one involving AWS SQS and storm depending on different httpclient libraries. Required downgrading the AWS SQS Java SDK version to 1.4.7 instead of the latest 1.7.2 at the time of posting.
It doesn't look like the storm-kestrel library works with the kestrel server anymore but someone else can correct if the understanding is wrong.
Used pykestrel for Python client and it works great.
Used simple-kestrel-client for Java kestrel client (Storm spout) and it works great
Used Java SDK v1.4.7 for AWS SQS due to the conflict and it works great.

来源：https://stackoverflow.com/questions/21580323/java-client-library-for-reading-from-kestrel-server-queue-from-within-storm-spou

标签

apache-storm

kestrel