kafka-producer-api

问题 I'm trying to get a simple Kafka Consumer to work using the Java API v0.9.0.1. The kafka server I'm using is a docker container, also running version 0.9.0.1. Below is the consumer code: public class Consumer { public static void main(String[] args) throws IOException { KafkaConsumer<String, String> consumer; try (InputStream props = Resources.getResource("consumer.props").openStream()) { Properties properties = new Properties(); properties.load(props); consumer = new KafkaConsumer<>

问题 I'm bit confused with kafka ACL configuration, where we configure authorization for producer and consumer. There are various examples showing producing/consuming message using command line. Do we need any extra configuration to produce/consume messages using JAVA api to/from secured kafka topic. 回答1: @Apollo : This question is quite vague.. If you want to learn ACL/ SSL it will take some time.. the below link might help you to get started. https://github.com/Symantec/kafka-security-0.9 回答2:

问题 I'm using program variables to create configuration objects and loading the schema from a local path which also been registered in kafka. Creating data object and using "Generic Record" method of serializing. var logMessageSchema =(Avro.RecordSchema)Avro.Schema.Parse(File.ReadAllText(@"C:\StatusMessageSchema\FileStatusMessageSchema.txt")); var record = new GenericRecord(logMessageSchema); record.Add("SystemID", "100"); record.Add("FileName", "ABS_DHCS"); record.Add("FileStatus", "3009");

问题 I'm using program variables to create configuration objects and loading the schema from a local path which also been registered in kafka. Creating data object and using "Generic Record" method of serializing. var logMessageSchema =(Avro.RecordSchema)Avro.Schema.Parse(File.ReadAllText(@"C:\StatusMessageSchema\FileStatusMessageSchema.txt")); var record = new GenericRecord(logMessageSchema); record.Add("SystemID", "100"); record.Add("FileName", "ABS_DHCS"); record.Add("FileStatus", "3009");

问题 I have a single node Kafka broker running inside a pod on a single node kubernetes environment. I am using this image for kafka: https://hub.docker.com/r/wurstmeister/kafka kafka version = 1.1.0 Kubernetes cluster is running inside a VM on a server. The VM has the following IP on the active interface ens32 - 192.168.3.102 Kafka.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: casb-deployment name: kafkaservice spec: replicas: 1 template: metadata: labels: app:

问题 When I am using the Java API of Kafka,if I let my main Thread sleep less than 2000ns,it cannot prodece any message.I really want to know why this happen? Here is my producer: public class Producer { private final KafkaProducer<String, String> producer; private final String topic; public Producer(String topic, String[] args) { //...... //...... producer = new KafkaProducer<>(props); this.topic = topic; } public void producerMsg() throws InterruptedException { String data = "Apache Storm is a

问题 I have this java code trying to add messages to a kafka queue String msgID = UUID.randomUUID().toString(); Properties prop = new Properties(); prop.put("metadata.broker.list", DEFAULT_BROKER); prop.put("serializer.class", "kafka.serializer.StringEncoder"); prop.put("request.required.acks", "-1"); prop.put("producer.type", "async"); ProducerConfig config = new ProducerConfig(prop); Producer<String, String> producer = new Producer<String, String>(config); KeyedMessage<String, String> message =

问题 I'm trying to achieve a transaction in a Kafka Processor to make sure I don't reprocess the same message twice. Given a message (A) I need to create a list of messages that will be produced on another topic in a transaction and i want to commit the original message (A) in the same transaction. From the documentation I found the Producer method sendOffsetsToTransaction which seems to be able to commit an offset in a transaction only if it succeeds. This is the code inside the process() method

问题 I am just starting with Kafka, it sounds really good for Microservices, but I work essentially in Scala. I added kafka to my sbt project with this: libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0" Then I do this: import org.apache.kafka.clients.producer.{Callback,KafkaProducer, Producer} ... val producer = new KafkaProducer[String, String](props) val record = new ProducerRecord[String, String]("my-topic", "key", "value") val fut = producer.send(record, callBack) ... My problem

问题 I'm running rdkafka_simple_producer.c to produce messages to a Kafka cluster. I have one topic and 30 partitions. Using the default round-robin partitioner. While the producer is working and generating messages to Kafka, I add more partitions to Kafka kafka/bin/kafka-topics.sh --alter --zookeeper server2:2181 --topic demotest --partitions 40 I'd expect the producer to realize about the change and eventually begin to produce to all 40 topics. However, at the end I only see data was produced to