kafka-producer-api

问题 I have an issue about sending message to Kafka using port forwarding. We are using port forwarding for Kafka like this: | Hostname | IP | Port | Port Forwarding | |----------|---------------|------|-----------------------| | kafka01 | 192.168.0.100 | 9092 | 106.107.118.119:30124 | | kafka02 | 192.168.0.101 | 9092 | 106.107.118.119:30125 | | kafka03 | 192.168.0.102 | 9092 | 106.107.118.119:30126 | To connect to kafka broker from my localhost, I have to use VPN to connect to these addresses 192

问题 I wrote a simple producer using pykafka but can't seem to get it to perform. The basic producer and call to produce is below. When I call this 100 times with a small message, and add some timing/profiling code, it takes about 14 seconds. I understand this to be an asynchronous sending of messages so I would expect it to be incredibly fast. Is there some setting I'm missing? I've also tried it with min_queued_messages=1 and those takes about 2 seconds longer. from pykafka import KafkaClient

问题 If I set Kafka config param at Producer as: 1. retries = 3 2. max.in.flight.requests.per.connection = 5 then its likely that Messages within one partition may not be in send_order. Does Kafka takes any extra step to make sure that messages within a partition remains in sent order only OR With above configuration, its possible to have out of order messages within a partition ? 回答1: Unfortunately, no. With your current configuration, there is a chance message will arrive unordered because of

问题 Closed . This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Update the question so it focuses on one problem only by editing this post. Closed 3 months ago . I have a couple questions in Kafka. 1) Does Kafka have a default web UI? 2) How can we gracefully shutdown a standalone kafka server, kafka console- consumer/console-producer. Any solutions will be highly appreciated. Thank you. 回答1: 1) No Kafka does not have a default UI. There

问题 Hy, I'm using spring-kafka 1.3.0.RELEASE to create a transactional producer. When the bootstrap server is down, the DefaultKafkaProducerFactory waits endlessly until the bootstrap server is up. What am I doing wrong ? Can I set a timeout and/or other similar properties ? This is an example of my code to reproduce the scenario: public static void main(String[] args) { final DefaultKafkaProducerFactory<Object, Object> producerFactory = new DefaultKafkaProducerFactory<>(producerConfigs());

问题 I have built a small testing environment on my Windows PC and write down the following code for testing kafka (using kafka_2.10:0.9.0.1 from org.apache.kafka). package iii.functiontesting; import java.text.ParseException; import java.util.Properties; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; /** * Hello world! * */ public class test4 { public static void main( String[] args ) throws ParseException { Properties

问题 I am running a Kafka producer in a local machine using my Intellij IDE & the producer will be producing a million records. While doing so, I want to capture the producer metrics in the below way: I am aware about JMX port for kafka & I did try setting the Kafka JMX port to 9999. But I am not sure if we can get the metrics using JConsole or JVisualVM in the above way that I am expecting. Can any one suggest any idea as to how can this be achieved? 回答1: In addition of JMX, the official Kafka

问题 EDIT2: Finally I have made my own producer using Java and it works well, so the problem is in the Kafka-console-producer . The kafka-console-consumer works well. EDIT: I have already tried with the version 0.9.0.1 and has the same behaviour. I am working on my bachelor's final project, a comparison between Spark Streaming and Flink. Before both frameworks I am using Kafka and a script to generate the data (explained below). My first test is to compare the latency between both frameworks with

问题 Kafka allows for asynchronous message sending through below methods on Producer (KafkaProducer) class: public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record) public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record, Callback callback) Successes can be handled through 1) the Future<RecordMetaData> object or 2) onCompletion method invoked by the callback. Full method signature and usage of onCompletion is as below (taken from kafka docs) `

问题 Kafka allows for asynchronous message sending through below methods on Producer (KafkaProducer) class: public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record) public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record, Callback callback) Successes can be handled through 1) the Future<RecordMetaData> object or 2) onCompletion method invoked by the callback. Full method signature and usage of onCompletion is as below (taken from kafka docs) `