kafka-producer-api

问题 With the last version of Kafka 0.11.0.0 the Apache team is introducing idempotent producer and transactions. Is It possible to guarantee that an entire set of messages (for example 1 million) we want to log, will be committed only at the end? I would like that, if for example the Producers loose the connection with the brokers and cannot restabilish it, no messages will be seen by the consumers. Is it possible? 回答1: Yes this is possible using Transactions in your producer. You start a

问题 I have a classic microservice architecture. So, there are differ applications. Each application may have 1..N instances. The system is deployed to Kubernetes. So, we have many differ PODs , which can start and stop in any time. I want to implement read-process-write pattern, so I need Kafka transactions. To configure transactions, I need to set some transaction id for each Kafka producer. (Actually, I need transaction-id-prefix , because of I use Spring for my applications, and it has such

问题 I have a Kafka Streams Application which takes data from few topics and joins the data and puts it in another topic. Kafka Configuration: 5 kafka brokers Kafka Topics - 15 partitions and 3 replication factor. Note: I am running Kafka Streams Applications on the same machines where my Kafka Brokers are running. Few millions of records are consumed/produced every hour. Whenever I take any kafka broker down, it goes into rebalancing and it takes approx. 30 minutes or sometimes even more for

问题 I don't see any failure while producing or consuming the data however there are bunch of duplicate messages in production. For a small topic which gets around 100k messages, there are ~4k duplicates though like I said there is no failure and on top of that there is no retry logic implemented or config value is set. I also check offset values for those duplicate messages and each has distinct values which tells me that the issue is in producer. Any help would be highly appreciated 回答1: Read

问题 I am trying to produce some messages to a single topic having 2 partitions. All the messages are going to partition number 2 only. I would expect that a producer stream would distribute the messages across all partitions. const kafka = require('kafka-node') const { Transform } = require('stream'); const _ = require('lodash'); const client = new kafka.KafkaClient({ kafkaHost: 'localhost:9092' }) , streamproducer = new kafka.ProducerStream({kafkaClient: client}); const stdinTransform = new

问题 I am trying to produce some messages to a single topic having 2 partitions. All the messages are going to partition number 2 only. I would expect that a producer stream would distribute the messages across all partitions. const kafka = require('kafka-node') const { Transform } = require('stream'); const _ = require('lodash'); const client = new kafka.KafkaClient({ kafkaHost: 'localhost:9092' }) , streamproducer = new kafka.ProducerStream({kafkaClient: client}); const stdinTransform = new

问题 I have installed Kafka 1.1.0 with Zookeeper 3.4.12 under Windows SubSystem for Linux on my Windows 10 laptop. I'm able to procude and consume messages while I stay in ubuntu but when I want to produce a message from windows (with a java program or with the tool kafka-console-producer.bat) I have the following error : [2018-05-11 15:31:01,449] ERROR Error when sending message to topic test with key: null, value: 15 bytes with error: (org.apache.kafka.clients.producer.internals

问题 I am using kafka 0.8.1.1. I have a 3 node kafka cluster with some topics having around 5 partitions. I planned to increase the number of nodes to 5 in cluster and moving some partitions from existing topics to the new brokers. Previous partition state: broker1 : topic1 { partition 0 } broker2 : topic1 { partition 1,2} broker3 : topic1 { partition 3,4} New intended state: broker1 : topic1 { partition 0} broker2 : topic1 { partition 1} broker3 : topic1 { partition 3} broker4 : topic1 {

问题 I have a use case where I have a JSON and I want to generate schema and record out of the JSON and publish a record. I have configured the value serializer and Schema setting is Backward compatible. First JSON String json = "{\n" + " \"id\": 1,\n" + " \"name\": \"Headphones\",\n" + " \"price\": 1250.0,\n" + " \"tags\": [\"home\", \"green\"]\n" + "}\n" ; Version 1 schema registered. Received message in avro console consumer. Second JSON. String json = "{\n" + " \"id\": 1,\n" + " \"price\":

问题 I have a use case where I have a JSON and I want to generate schema and record out of the JSON and publish a record. I have configured the value serializer and Schema setting is Backward compatible. First JSON String json = "{\n" + " \"id\": 1,\n" + " \"name\": \"Headphones\",\n" + " \"price\": 1250.0,\n" + " \"tags\": [\"home\", \"green\"]\n" + "}\n" ; Version 1 schema registered. Received message in avro console consumer. Second JSON. String json = "{\n" + " \"id\": 1,\n" + " \"price\":