apache-kafka

问题 Is there a way to check the current total memory usage of all the messages in topic? We have a retention policy set to a max number of bytes and we're trying to check if we are actually hitting max or not. We see that some messages are missing from the topic. We believe it might be because we hit the max size. 来源： https://stackoverflow.com/questions/62582720/how-to-check-kafka-topic-memory-usage

问题 I am trying to test a topology that, as the last node, has a KTable. My test is using a full-blown Kafka Cluster (through confluent's Docker images) so I am not using the TopologyTestDriver . My topology has input of key-value types String -> Customer and output of String -> CustomerMapped . The serdes, schemas and integration with Schema Registry all work as expected. I am using Scala, Kafka 2.2.0, Confluent Platform 5.2.1 and kafka-streams-scala . My topology, as simplified as possible,

问题 I am trying to test a topology that, as the last node, has a KTable. My test is using a full-blown Kafka Cluster (through confluent's Docker images) so I am not using the TopologyTestDriver . My topology has input of key-value types String -> Customer and output of String -> CustomerMapped . The serdes, schemas and integration with Schema Registry all work as expected. I am using Scala, Kafka 2.2.0, Confluent Platform 5.2.1 and kafka-streams-scala . My topology, as simplified as possible,

问题 I am trying to test a topology that, as the last node, has a KTable. My test is using a full-blown Kafka Cluster (through confluent's Docker images) so I am not using the TopologyTestDriver . My topology has input of key-value types String -> Customer and output of String -> CustomerMapped . The serdes, schemas and integration with Schema Registry all work as expected. I am using Scala, Kafka 2.2.0, Confluent Platform 5.2.1 and kafka-streams-scala . My topology, as simplified as possible,

问题 I’m new to Kafka and trying out few small usecase for my new application. The use case is basically, Kafka-producer —> Kafka-Consumer—> flume-Kafka source—>flume-hdfs-sink. When Consuming(step2), below is the sequence of steps.. 1. consumer.Poll(1.0) 1.a. Produce to multiple topics (multiple flume agents are listening) 1.b. Produce. Poll() 2. Flush() every 25 msgs 3. Commit() every msgs (asynchCommit=false) Question 1: Is this sequence of action right!?! Question2: Will this cause any data

问题 I find that the __consumer_offsets topic log size is growing rapidly and after studying it further found the topics with the highest volume. I changed the retention policy on these topics to stop the rate of growth but would like to increase disk space and delete all the old logs for __consumer_offsets topic. But this will cause all the other topics and consumers/producers to get corrupted or lose valuable metadata. Is there a way I can accomplish this? I'm looking at the parameters for the

问题 I understand that the default TTL is set to infinity (non-positive). However, if we need to retain data in the store for max of 2 days, can we do the override with the RocksDBConfigSetter interface implementation, that is options.setWalTtlSeconds(172800)? OR would it conflict with the Kafka streams internals? Ref: https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config 回答1: This is currently not possible. Kafka Streams disables

问题 I understand that the default TTL is set to infinity (non-positive). However, if we need to retain data in the store for max of 2 days, can we do the override with the RocksDBConfigSetter interface implementation, that is options.setWalTtlSeconds(172800)? OR would it conflict with the Kafka streams internals? Ref: https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config 回答1: This is currently not possible. Kafka Streams disables

问题 Is it possible to run complete Confluent Platform on Windows ? If not what is preferable way to run Confluent platform ? Thanks 回答1: Per list of supported operating systems: Windows is not currently supported. Windows users can download and use the ZIP and TAR archives, but must run the JAR files directly. You could also use Docker to run it. 回答2: In addition to Docker on Windows 10 Pro you can install Confluent Platform on Windows 10 Subsystem for Linux 来源： https://stackoverflow.com

问题 Is it possible to run complete Confluent Platform on Windows ? If not what is preferable way to run Confluent platform ? Thanks 回答1: Per list of supported operating systems: Windows is not currently supported. Windows users can download and use the ZIP and TAR archives, but must run the JAR files directly. You could also use Docker to run it. 回答2: In addition to Docker on Windows 10 Pro you can install Confluent Platform on Windows 10 Subsystem for Linux 来源： https://stackoverflow.com