apache-kafka

问题 I am using Apache Kafka. I use KafkaProducer to produce data and KafkaConsumer to consume data. My config data is: Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ProducerConfig.CLIENT_ID_CONFIG, "DemoProducer"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.IntegerSerializer"); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization

问题 I want to get response object from Kafka consumer. Full code In my case as you can see I'm casting input to a factory object instead of using the factory object to transform the input into custom object. ConsumerRecord<String, SaleResponseFactory> consumerRecord = replyFuture.get(10, TimeUnit.SECONDS); SaleResponseFactory value = (SaleResponseFactory) consumerRecord.value(); System.out.println("!!! " + value.getUnique_id()); It's not very clear for me how I can use the factory pattern in this

问题 If I have a Kafka stream application that fails to post to a topic (because the topic does not exist) does it commit the consumer offset and continue, or will it loop on the same message until it can resolve the output topic? The application merely prints an error and runs fine otherwise from what I can observe. An example of the error when trying to post to topic: Error while fetching metadata with correlation id 80 : {super.cool.test.topic=UNKNOWN_TOPIC_OR_PARTITION} In my mind it would

问题 We currently do a real-time aggregation of data in an event-store. The idea is to visualize transaction data for multiple time ranges (monthly, weekly, daily, hourly) and for multiple nominal keys. We regularly have late data, so we need to account for that. Furthermore the requirement is to display "running" results, that is value of the current window even before it is complete. Currently we are using Kafka and Apache Storm (specifically Trident i.e. microbatches) to do this. Our

问题 We currently do a real-time aggregation of data in an event-store. The idea is to visualize transaction data for multiple time ranges (monthly, weekly, daily, hourly) and for multiple nominal keys. We regularly have late data, so we need to account for that. Furthermore the requirement is to display "running" results, that is value of the current window even before it is complete. Currently we are using Kafka and Apache Storm (specifically Trident i.e. microbatches) to do this. Our

问题 We currently do a real-time aggregation of data in an event-store. The idea is to visualize transaction data for multiple time ranges (monthly, weekly, daily, hourly) and for multiple nominal keys. We regularly have late data, so we need to account for that. Furthermore the requirement is to display "running" results, that is value of the current window even before it is complete. Currently we are using Kafka and Apache Storm (specifically Trident i.e. microbatches) to do this. Our

问题 I read a ton of Gary Russell answers and posts, but didn't find actual solution for the common use-case for synchronization of the sequence below: recieve from topic A => save to DB via Spring-data => send to topic B As i understand properly: there is no guarantee for fully atomic processing in that case and i need to deal with messages deduplication on the client side, but the main issue is that ChainedKafkaTransactionManager doesn't synchronize with JpaTransactionManager (see @KafkaListener

问题 I am having difficulties creating a basic spark streaming application. Right now, am trying it on my local machine. I have done following setup. -Setup Zookeeper -Setup Kafka ( Version : kafka_2.10-0.9.0.1) -Created a topic using below command kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test -Started producer and consumer on two different cmd terminals using below commands Producer : kafka-console-producer.bat --broker-list localhost:9092

问题 I am a trying to integrate Storm with Kafka. I am running a Storm topology using KafkaSpout . This is the basic word-count topology. I am using Apache Kafka as the source and storm for processing the data. While submitting the topology I am facing these issues. I am very new to Kafka and Storm. Please suggest me the changes I need to do in the following code. This is my code: public class TopologyMain { private static final String SENTENCE_SPOUT_ID = "kafka-sentence-spout"; private static

问题 Is there a way to check the current total memory usage of all the messages in topic? We have a retention policy set to a max number of bytes and we're trying to check if we are actually hitting max or not. We see that some messages are missing from the topic. We believe it might be because we hit the max size. 来源： https://stackoverflow.com/questions/62582720/how-to-check-kafka-topic-memory-usage