kafka-consumer-api

问题 max.poll.records has been recently changed to 500 in kafka consumer config, but I am wondering how does this affect the consumer poll. Is it just an upper bound of the maximum number of records that could be fetched or the consumer waits till it gets 500 records. 回答1: max.poll.records : Yes from new consumer this property is changed to 500 by default which means consumer can poll minimum 1 to max 500 records for each poll, and which means consumer will not wait when partition did not have

问题 I am trying to see if the error channel I am passing to @ServiceActivator can be bounded/parameterized referring the value specified in YAML instead of hardcoding actual destination and consumer group in the code itself. @ServiceActivator( // I do not want to hardcode destination and consumer group here inputChannel = "stream-test-topic.my-consumer-group.errors" ) public void handleError(ErrorMessage errorMessage) { // Getting exception objects Throwable errorMessagePayload = errorMessage

问题 I am trying to see if the error channel I am passing to @ServiceActivator can be bounded/parameterized referring the value specified in YAML instead of hardcoding actual destination and consumer group in the code itself. @ServiceActivator( // I do not want to hardcode destination and consumer group here inputChannel = "stream-test-topic.my-consumer-group.errors" ) public void handleError(ErrorMessage errorMessage) { // Getting exception objects Throwable errorMessagePayload = errorMessage

问题 Our kafka system crashed because no disk space was available. The consumers are Spring boot application which are using the Kafka Streams API. Now every consumer application shows the following error: java.io.FileNotFoundException: /tmp/kafka-streams/908a79bc-92e7-4f9c-a63a-5030cf4d3555/streams.device-identification-parser/0_48/.checkpoint.tmp (No such file or directory) This exception occurred exactly after the kafka server was restarted. If we restart the application, the service starts at

问题 In some cases, I use Kafka-stream to model a small in memory (hashmap) projection of a topic. The K,V cache does require some manipulations, so it is not a good case for a GlobalKTable. In such a “caching” scenario, I want all my sibling instances to have the same cache, so I need to bypass the consumer-group mechanism. To enable this, I normally simply start my apps with a randomly generated application Id, so each app will reload the topic each time it restarts. The only caveat to that is

问题 I am working on a application in which I am using kafka and tech is scala. My kafka consumer code is as follows: val props = new Properties() props.put("group.id", "test") props.put("bootstrap.servers", "localhost:9092") props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") props.put("auto.offset.reset", "earliest") props.put("group.id", "consumer-group") val

问题 From what I understand a consumer reads messages off a particular topic, and the consumer client will periodically commit the offset. So if for some reason the consumer fails a particular message, that offset won't be committed and you can then go back and reprocess he message. Is there anything that tracks the offset you just consumed and the offset you then commit? 回答1: Does kafka distinguish between consumed offset and commited offset? Yes, there is a big difference. The consumed offset is

问题 Hello I am writing a kafka consumer-producer using spring cloud stream . Inside my consumer I save my data to a database , if the database goes down I will exit the application manually .After restarting application if the database is still down as a result the application gets stopped again . Now if i restart the application for the third time the messages received in the middle interval(the two failures) are lost, kafka consumer takes the latest message , also it skips the message on which

问题 I have two Kafka (2.11-0.11.0.1) brokers. Default replication factor of topics is set to 2. Producers write data only to zero partition. And I have scheduled executor which runs the task periodically. When it consumes a topic with a small number of records per minute (100 per minute) then in works like a charm. But for huge topics (10K per minute) method poll returns no data. The task is: import org.apache.kafka.clients.consumer.Consumer; import org.apache.kafka.clients.consumer

问题 We have java app which runs on app server. We have topic topic1 with 5 partitions. We have created 5 consumer threads in a consumer group "consumergroup1". One consumer thread per partition. They are running on appserver server one. Now we bring up one more app server servertwo on another node. We want to run some of consumers in same group "consumergroup1" for topic1 on new server. Basically distribute consumer threads between two app servers for same consumer group. If we bring up consumers