apache-kafka

问题 Found these two questions : here and here, but I still don't quite understand. I still got (unexpected?) behaviour. I try to log-compact kafka topic using this configuration kafka-topics.sh --bootstrap-server localhost:9092 --create --partitions 1 --replication-factor 1 --topic test1 --config "cleanup.policy=compact" --config "delete.retention.ms=1000" --config "segment.ms=1000" --config "min.cleanable.dirty.ratio=0.01" --config "min.compaction.lag.ms=500" Then I send these messages, each has

问题 I'm trying to secure the REST endpoint in kafka connect with https. Below is my config in connect.distributed.properties file, I created self-signed certificate in my Linux VM, listeners=https://myhostname.xxxxx.xx.com:8085 listeners.https.ssl.keystore.location=/home/kafka/server.keystore.jks listeners.https.ssl.keystore.password=******* listeners.https.ssl.key.password=****** listeners.https.ssl.truststore.location=/home/kafka/server.truststore.jks listeners.https.ssl.truststore.password=***

问题 I'm thinking about the following use-case and would like to validate if this approach is conceptually valid. The goal is to expose a long-running Server-Sent-Event (SSE) endpoint in Spring replaying the same Kafka topic for each incoming connection (with some user-specific filtering). The SSE is exposed in this way: @GetMapping("/sse") public SseEmitter sse() { SseEmitter sseEmitter = new SseEmitter(); Executors .newSingleThreadExecutor() .execute(() -> dummyDataProducer.generate() // kafka

问题 Am sending the CSV data to Kafka topic using Kafka-Python . Data is sent and received by Consumer successfully. Now am trying to stream a csv file continuously, any new entry added to the file should be automatically sent to Kafka topic. Any suggestion would be helpful on continuous streaming of CSV file Below is my existing code, from kafka import KafkaProducer import logging from json import dumps, loads import csv logging.basicConfig(level=logging.INFO) producer = KafkaProducer(bootstrap

问题 i connect to server through ssh, i launch my zookeper kafka, and my debezium connector, after a while only the kafka terminal tab get's kicked out with the following error packet_write_wait: Connection to **.**.***.*** port 22: Broken pipe and my connector output is: >>>>[2019-07-10 10:04:49,563] WARN [Producer clientId=producer-1] >>>>Connection to node 0 (ip-***.**.**.***.eu- >>>>west-3.compute.internal/***.**.**.***:9092) could not be established. >>>>Broker may not be available. >>>>(org

问题 i connect to server through ssh, i launch my zookeper kafka, and my debezium connector, after a while only the kafka terminal tab get's kicked out with the following error packet_write_wait: Connection to **.**.***.*** port 22: Broken pipe and my connector output is: >>>>[2019-07-10 10:04:49,563] WARN [Producer clientId=producer-1] >>>>Connection to node 0 (ip-***.**.**.***.eu- >>>>west-3.compute.internal/***.**.**.***:9092) could not be established. >>>>Broker may not be available. >>>>(org

问题 I setup ACL for 3 node Kafka cluster and able to send and receive for a topic through producer console and consumer console. Now I want to configure Kafka connect with ACL. I tried with SASL_PLAINTEXT combinations and in connect.log file it shows the following error. it is not syncing to from source table to topic, please help where I am missing any configuration. error log [2020-10-14 07:24:35,874] ERROR WorkerSourceTask{id=oracle-jdbc-source-mtx_domains_acl5-0} Failed to flush, timed out

问题 I would like to understand how the communication between the Kafka and Spark(Streaming) nodes takes place. I have the following questions. If Kafka servers and Spark nodes are in two separate clusters how would be communications takes place. What are the steps need to configure them. If both are in same clusters but are in different nodes, how will be communication happens. communication i mean here is whether it is a RPC or Socket communication. I would like to understand the internal

问题 I would like to understand how the communication between the Kafka and Spark(Streaming) nodes takes place. I have the following questions. If Kafka servers and Spark nodes are in two separate clusters how would be communications takes place. What are the steps need to configure them. If both are in same clusters but are in different nodes, how will be communication happens. communication i mean here is whether it is a RPC or Socket communication. I would like to understand the internal

问题 Current setup - I am streaming data from 16 single partitioned topics and doing KTable-KTable joins and sending an output with aggregated data from all streams. I am also materializing each KTable to local state-store. Scenarios - When I tried running two app instances, I was expecting it kafka streams to run on single instance but for some reason it ran on other instance too. Looks like it can created stream task on other app instance during kafka streams failure on instance#1 during to some