kafka-python consumer start reading from offset (automatically)

Question

I\'m trying to build an application with kafka-python where a consumer reads data from a range of topics. It is extremely important that the consumer never reads the same messag

Answer 1

You are getting this behavior because your consumer is not using a Consumer Group. With a Consumer Group, the consumer will regularly commit (save) its position to Kafka. That way if it's restarted it will pick up from its last committed position.

To make your consumer use a Consumer Group, you need to set group_id when constructing it. See group_id description from the docs:

The name of the consumer group to join for dynamic partition assignment (if enabled), and to use for fetching and committing offsets. If None, auto-partition assignment (via group coordinator) and offset commits are disabled. Default: None

For example:

consumer = KafkaConsumer('numtest', bootstrap_servers=['localhost:9092'],
                         auto_offset_reset='earliest', enable_auto_commit=True,
                         auto_commit_interval_ms=1000, group_id='my-group')

Answer 2

Is that possible to using consumer from different server. I already tried the same below is the code and its not fetching any data from kafka.

consumer = KafkaConsumer('tet', bootstrap_servers=['192.168.1.20:9092'],
                     auto_offset_reset='earliest', enable_auto_commit=True,
                     auto_commit_interval_ms=1000, group_id=None)

Note:- When I am giving wrong ip or port number its throws exceptions.