Consuming a kinesis stream in python

Question

I cant seem to find a decent example that shows how can I consume an AWS Kinesis stream via Python. Can someone please provide me with some examples I could look into?

B

Answer 1

While this question has already been answered, it might be a good idea for future readers to consider using the Kinesis Client Library (KCL) for Python instead of using boto directly. It simplifies consuming from the stream when you have multiple consumer instances, and/or changing shard configurations.

https://aws.amazon.com/blogs/aws/speak-to-kinesis-in-python/

A more complete enumeration of what the KCL provides

• Connects to the stream
• Enumerates the shards
• Coordinates shard associations with other workers (if any)
• Instantiates a record processor for every shard it manages
• Pulls data records from the stream
• Pushes the records to the corresponding record processor
• Checkpoints processed records (it uses DynamoDB so your code doesn't have to manually persist the checkpoint value)
• Balances shard-worker associations when the worker instance count changes
• Balances shard-worker associations when shards are split or merged

The items in bold are the ones that I think are where the KCL really provides non-trivial value over boto. But depending on your usecase boto may be much much much simpler.