Kafka consumer - what's the relation of consumer processes and threads with topic partitions

后端 未结 3 1161
被撕碎了的回忆
被撕碎了的回忆 2021-02-02 03:48

I have been working with Kafka lately and have bit of confusion regarding the consumers under a consumer group. The center of the confusion is whether to implement consumers as

3条回答
  •  逝去的感伤
    2021-02-02 04:31

    Thanks for the detailed answer from @user2720864, but I think the re-allocation case @user2720864 mentioned in the answer is not correct => one partition cannot be consumed by two consumers.

    When there are more consumers (compared to the partitions), each partition will be exclusively allocated to one consumer only while the leftover consumers will stay lazy only until some working consumers being dead or being removed from the group.

    Based on the Kafka Consumers document:

    The way consumption is implemented in Kafka is by dividing up the partitions in the log over the consumer instances so that each instance is the exclusive consumer of a "fair share" of partitions at any point in time. This process of maintaining membership in the group is handled by the Kafka protocol dynamically. If new instances join the group they will take over some partitions from other members of the group; if an instance dies, its partitions will be distributed to the remaining instances.

    And also its API specification at "Consumer Groups and Topic Subscriptions" section:

    This is achieved by balancing the partitions between all members in the consumer group so that each partition is assigned to exactly one consumer in the group.

提交回复
热议问题