apache-kafka

Indefinite log retention on kafka

最后都变了- 提交于 2021-02-07 08:18:42
问题 I am using kafka for event-sourcing. I realized that we still need to configure the log retention time, i.e. log.retention.hours . What is the best value to use if I want to keep all my messages indefinitely? The sample configuration for log.retention.bytes is set to -1, can I use -1 also in the log.retention.hours ? 回答1: See the following Kafka JIRA which is due for the 0.9.0.0 release. For the time being set as suggested: log.retention.bytes = -1 log.retention.hours = 2147483647 Which is

Indefinite log retention on kafka

≯℡__Kan透↙ 提交于 2021-02-07 08:14:00
问题 I am using kafka for event-sourcing. I realized that we still need to configure the log retention time, i.e. log.retention.hours . What is the best value to use if I want to keep all my messages indefinitely? The sample configuration for log.retention.bytes is set to -1, can I use -1 also in the log.retention.hours ? 回答1: See the following Kafka JIRA which is due for the 0.9.0.0 release. For the time being set as suggested: log.retention.bytes = -1 log.retention.hours = 2147483647 Which is

Object not serializable (org.apache.kafka.clients.consumer.ConsumerRecord) in Java spark kafka streaming

十年热恋 提交于 2021-02-07 07:10:22
问题 I am pretty sure that I am pushing data only string and deserialize also as String. Record I pushed it is showing in error also. But why suddenly it is showing such type of error, Is there anything I am missing? Here is below code, import java.util.HashMap; import java.util.HashSet; import java.util.Arrays; import java.util.Collection; import java.util.Iterator; import java.util.Map; import java.util.Set; import java.util.concurrent.atomic.AtomicReference; import java.util.regex.Pattern;

Object not serializable (org.apache.kafka.clients.consumer.ConsumerRecord) in Java spark kafka streaming

时光总嘲笑我的痴心妄想 提交于 2021-02-07 07:08:41
问题 I am pretty sure that I am pushing data only string and deserialize also as String. Record I pushed it is showing in error also. But why suddenly it is showing such type of error, Is there anything I am missing? Here is below code, import java.util.HashMap; import java.util.HashSet; import java.util.Arrays; import java.util.Collection; import java.util.Iterator; import java.util.Map; import java.util.Set; import java.util.concurrent.atomic.AtomicReference; import java.util.regex.Pattern;

Object not serializable (org.apache.kafka.clients.consumer.ConsumerRecord) in Java spark kafka streaming

纵饮孤独 提交于 2021-02-07 07:07:27
问题 I am pretty sure that I am pushing data only string and deserialize also as String. Record I pushed it is showing in error also. But why suddenly it is showing such type of error, Is there anything I am missing? Here is below code, import java.util.HashMap; import java.util.HashSet; import java.util.Arrays; import java.util.Collection; import java.util.Iterator; import java.util.Map; import java.util.Set; import java.util.concurrent.atomic.AtomicReference; import java.util.regex.Pattern;

Sending Apache Kafka data on web page

余生长醉 提交于 2021-02-07 06:55:49
问题 I am building a real time energy monitoring system, where the data are from sensors. There will be new data every second. The data used will be aggregated to be rendered as charts. I have looked into real time stream processing with big amounts of data, and it lead me to Apache Kafka. Right now my web app is using Express js. I am using kafka-node library. Now currently, I manually insert new data through the command line as a producer . In my server code, I have set-up a consumer that

Sending Apache Kafka data on web page

梦想与她 提交于 2021-02-07 06:55:47
问题 I am building a real time energy monitoring system, where the data are from sensors. There will be new data every second. The data used will be aggregated to be rendered as charts. I have looked into real time stream processing with big amounts of data, and it lead me to Apache Kafka. Right now my web app is using Express js. I am using kafka-node library. Now currently, I manually insert new data through the command line as a producer . In my server code, I have set-up a consumer that

Kafka consumer, very long rebalances

不羁的心 提交于 2021-02-07 05:41:57
问题 We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java app which spawns many consumer threads consuming from different topics. For every topic we have specified different consumer-group. A lot of times I see that whenever this application is restarted one or more CGs take more than 5 minutes to receive partition assignment. Till that time consumers for that topic don't consume anything. If I go to Kafka broker and run consumer-groups.sh and describe that particular CG I see that it

Real time notification with Kafka and NodeJS

时间秒杀一切 提交于 2021-02-06 12:56:12
问题 In my project, I have to design a real time notification system. And I did it as the image below. You can see that I used Kafka as a queue messaging system, and NodeJS to build Websocket Server and Kafka Consumers. Producers will collect the notification data and push it to Kafka. Consumers will read and process the data from Kafka and push it to client via websocket if that data belongs to that user. With the architect above, Each online user will open a Websocket connection and create a new

Why is there inconsistency in Kafka's ordering guarantees when using Idempotent Producer?

妖精的绣舞 提交于 2021-02-06 11:27:26
问题 I am using Kafka 1.0.1 in my application and I have started using the Idempotent Producer feature that was introduced in 0.11, and I've having trouble understanding the ordering guarantees when using the Idempontent feature. My producer's configuration is: enable.idempotence = true max.in.flight.requests.per.connection = 5 retries = 50 acks = all According to the documentation: retries Setting a value greater than zero will cause the client to resend any record whose send fails with a