kafka-producer-api

问题 How should I know when i have to scale the consumer in consumer group . What are the triggers for the consumers to scale when there is a fast producer ? 回答1: In Kafka while creating a topic, need to provide number of partitions and replication factor . Let say there is one topic called TEST with 10 partitions, for parallel consumption of data need to create consumer group with 10 consumers, where each consumer will be consuming the data from the respective partition. Here is the catch, if the

问题 When we produce messages we can define a callback, this callback can expect an exception: kafkaProducer.send(producerRecord, new Callback() { public void onCompletion(RecordMetadata recordMetadata, Exception e) { if (e == null) { // OK } else { // NOT OK } } }); Considered the buitl-in retry logic in the producer, I wonder which kind of exception should developers deal explicitly with? 回答1: According to the Callback Java Docs there are the following Exception possible happening during

问题 In my Scala (2.11) stream application I am consuming data from one queue in IBM MQ and writing it to a Kafka topic that has one partition. After consuming the data from the MQ the message payload gets splitted into 3000 smaller messages that are stored in a Sequence of Strings. Then each of these 3000 messages are send to Kafka (version 2.x) using KafkaProducer. How would you send those 3000 messages? I can't increase the number of queues in IBM MQ (not under my control) nor the number of

问题 I am using Spring for Apache Kafka and have created a service that uses a Kafka Producer (org.apache.kafka.clients.producer) via Spring's KafkaTemplate to send messages to a topic. On the target Kafka cluster I have disabled auto topic creation. Using a combination of producer configurations listed here https://kafka.apache.org/documentation/#producerconfigs I am successfully controlling how many times a request is retried, time between retries, etc. If I provide a topic that does not exist

问题 I have a json payload which i want to send as header in producer Api { "type": "record_created", "version": 1, "orgId": "", "userId": "", "userName": "", "correlationId": "", "jobId": "" } Above payload should be sent as header producer.send(new ProducerRecord<Integer, String>(topic, messageNo, records,header) How can we do this ? 回答1: There are two ways 1) getHeaders and add Header ProducerRecord<Integer, String> record = new ProducerRecord<Integer, String>("topic", 1, "message"); record

问题 my sink properties : { "name": "jdbc-oracle", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "orders", "connection.url": "jdbc:oracle:thin:@10.1.2.3:1071/orac", "connection.user": "ersin", "connection.password": "ersin!", "auto.create": "true", "delete.enabled": "true", "pk.mode": "record_key", "pk.fields": "id", "insert.mode": "upsert", "plugin.path": "/home/ersin/confluent-5.4.1/share/java/", "name": "jdbc-oracle" }, "tasks": [ {

问题 my sink properties : { "name": "jdbc-oracle", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "orders", "connection.url": "jdbc:oracle:thin:@10.1.2.3:1071/orac", "connection.user": "ersin", "connection.password": "ersin!", "auto.create": "true", "delete.enabled": "true", "pk.mode": "record_key", "pk.fields": "id", "insert.mode": "upsert", "plugin.path": "/home/ersin/confluent-5.4.1/share/java/", "name": "jdbc-oracle" }, "tasks": [ {

问题 What's the difference between the following ways of enabling compression in kafka: Approach 1: Create a topic using the command: bin/kafka-topics.sh --create --zookeeper localhost:2181 --config compression.type=gzip --topic test Approach 2: Set the property compression.type = gzip in Kafka Producer Client API. I get better compression and higher throughput when using Approach 1. If I use Approach 1, does it mean that the compression occurs at the broker end while in Approach 2, the messages

问题 This question is about architecture and kafka topics migrating. Original problem : schema evolution without backward compatibility. https://docs.confluent.io/current/schema-registry/avro.html I am asking the community to give me an advice or share articles from which I can get inspired and maybe think of a solution to my problem. Maybe there is an architecture or streaming pattern. It is not necessary to give me a language specific solution; just give me a direction into which I can go... My

问题 I would like to have Exactly-once semantics, but I don't want to read message with Consumer. I'd rather read messages with Kafka Streams AP. If I add processing.guarantee=exactly_once to Stream configuration, will exactly-once semantics be kept? 回答1: Exactly-once processing is based on a read-process-write pattern. Kafka Streams uses this pattern and thus, if you write a regular Kafka Streams application that writes the result back to a Kafka topic, you will get exactly-once processing