apache-kafka

问题 In kafka documentation i'm trying to understand this property max.poll.interval.ms The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. This mean each poll will happen

问题 As I understand from the documentation of kafka connect this configuration should ignore the keys for metricbeat and filebeat topic but not for alarms. But kafka connect does not ignore any key. So that's the fully json config that i pushing to kafka-connect over rest { "auto.create.indices.at.start": false, "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "connection.url": "http://elasticsearch:9200", "connection.timeout.ms": 5000, "read.timeout.ms": 5000,

问题 I'm trying to understand databricks delta and thinking to do a POC using Kafka. Basically the plan is to consume data from Kafka and insert it to the databricks delta table. These are the steps that I did: Create a delta table on databricks. %sql CREATE TABLE hazriq_delta_trial2 ( value STRING ) USING delta LOCATION '/delta/hazriq_delta_trial2' Consume data from Kafka. import org.apache.spark.sql.types._ val kafkaBrokers = "broker1:port,broker2:port,broker3:port" val kafkaTopic = "kafkapoc"

问题 I'm not able to connect to a kafka server(first server I tried on WSL2) running on ubuntu, via intellij or vscode running on windows. I even tried using the VM's IP, but no luck. As I understand, we should be able to connect using 'localhost' as per this doc https://docs.microsoft.com/en-us/windows/wsl/compare-versions am I missing something? Here is my code Properties producerProperties = new Properties(); producerProperties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost

问题 I need to find or write an SMT that will add header fields to a request. The request is missing some type fields and I want to add them. How exactly do you add a header within an SMT all I have seen are just record transforms like below but what if its the header I want to change or add a field to? private R applySchemaless(R record) { final Map<String, Object> value = requireMap(operatingValue(record), PURPOSE); // record.headers.add(Header) but how do I define the header // or record

问题 I'm trying to connect my SQL Server database to kafka on windows, i downloaded Debezium jar files, they are in a folder named debezium-connector-sqlserver, here's what it looks like : Kafka folder, Debezium folder I also added this line plugin.path=C:\\current_kafka_2.12-2.0.0\\debezium-connector-sqlserver in connect-standalone.properties, it's where i put all the jar files i downloaded from Debezium. i also created a file named connect-mssql.properties, and i put this in it. name=inventory

问题 We are using kafka-streams 2.3.1 and I've just noticed that if broker is down, the streams app seems to be content to try to keep trying connecting forever. new KafkaStreams(createTopology(), properties()).start() o.apache.kafka.clients.NetworkClient - [AdminClient clientId=test] Connection to node -1 (broker/127.0.0.1:9092) could not be established. Broker may not be available. The streams state is REBALANCING while this is going on so there's no good way to determine if the connection is

问题 I've created a topic in kafka using kafka-topic.sh and tested it with java client: kafka-topics.sh \ --create \ --zookeeper localhost:2181 \ --replication-factor 1 \ --partitions 2 \ --topic my-topic KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props); consumer.subscribe(Arrays.asList("my-topic"), new LoggingConsumerRebalanceListener(RandomStringUtils.randomAlphanumeric(3).toLowerCase())); while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis

问题 I'm trying to implement a reply response pattern in which I publish a message to a topic listened to by several consumer groups. That means that they will all get the message as well as submit the response in the reply topic. The problem is because they all respond to the same message, only the first received message in the reply topic shall be answered. The others will be discarded. Given that I know how many responses I should be getting on the reply topic(call that number-n), how can I

问题 I'm trying to implement a reply response pattern in which I publish a message to a topic listened to by several consumer groups. That means that they will all get the message as well as submit the response in the reply topic. The problem is because they all respond to the same message, only the first received message in the reply topic shall be answered. The others will be discarded. Given that I know how many responses I should be getting on the reply topic(call that number-n), how can I