multiple consumers per kinesis shard

Question

I read you can have multiple consumer apps per kinesis stream.

http://docs.aws.amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.html

however, I h

Answer 1

Kinesis Client Library starts threads in the background, each listens to 1 shard in the stream. You cannot connect to a shard over multiple threads, that is by-design.

http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-scaling.html

For example, if your application is running on one EC2 instance, and is processing one Amazon Kinesis stream that has four shards. This one instance has one KCL worker and four record processors (one record processor for every shard). These four record processors run in parallel within the same process.

In the explanation above, the term "KCL worker" refers to a Kinesis consumer application. Not the threads.

But below, the same "KCL worker" term refers to a "Worker" thread in the application; which is a runnable.

Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards (except for failure standby purposes). Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard.

See the Worker.java class in KCL source.

Answer 2

Late to the party, but the answer is that you can have multiple consumers per kinesis shard. A KCL instance will only start one process per shard, but you can have another KCL instance consuming the same stream (and shard), assuming the second one has permission.

There are limits, though, as laid out in the docs, including:

Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second.

If you want a stream with multiple consumers where each message will be processed once, you're probably better off with something like Amazon Simple Queue Service.