confluent

问题 I'm following the instructions linked in this wiki doc to install the confluent platform on my EC2 instance running amazon linux (version 2016.09). I did everything it says including: $ sudo rpm --import http://packages.confluent.io/rpm/3.1/archive.key Created /etc/yum.repos.d/confluent.repo with these contents: [Confluent.dist] name=Confluent repository (dist) baseurl=http://packages.confluent.io/rpm/3.1/6 gpgcheck=1 gpgkey=http://packages.confluent.io/rpm/3.1/archive.key enabled=1

问题 Produce to/Consume from Kafka in JSON. Save to HDFS in JSON using below properties : key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=false value.converter.schemas.enable=false Producer : curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" \ --data '{"schema": {"type": "boolean", "optional": false, "name": "bool", "version": 2, "doc": "the documentation", "parameters": {"foo": "bar

问题 I was following step #7 (Use Kafka Connect to import/export data) at this link: http://kafka.apache.org/documentation.html#quickstart It was working well until I deleted the 'test.txt' file. Mainly because that's how log4j files would work. After certain time, the file will get rotated - I mean - it will be renamed & a new file with the same name will start getting written to. But after, I deleted 'test.txt', the connector stopped working. I restarted connector, broker, zookeeper etc, but the

问题 I'm trying to use KSQL (as part of confluent-5.0.0) to create a single record out of a set of parent records and child records, where every parent record has multiple child records (spefically, payment details and the parties involved in the payment). These parent/child records are linked by the parent's id. To illustrate, I'm dealing with records of roughly this structure in the source system: payment: | id | currency | amount | payment_date | |------------------------------------------| |

问题 I am trying to insert data into a postgres database from a topic in kafka. I am using the following command to load ./bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties etc/kafka-connect-jdbc/sink-quickstart-mysql.properties The sink-quickstart-mysql.properties is as follows name=test-sink-mysql-jdbc-autoincrement connector.class=io.confluent.connect.jdbc.JdbcSinkConnector tasks.max=1 topics=third_topic connection.url=jdbc:postgres://localhost:5432/postgres

问题 Confluent Kafka 5.0.0 has been installed on AWS EC2 which has Public IP say 54.XX.XX.XX Opened port 9092 on the EC2 machine with 0.0.0.0 In /etc/kafka/server.properties I have advertised.listeners=PLAINTEXT://54.XX.XX.XX:9092 listeners=PLAINTEXT://0.0.0.0:9092 In /etc/kafka/producer.properties I have bootstrap.servers=0.0.0.0:9092 on local machine In /etc/kafka/consumer.properties I have bootstrap.servers=54.XX.XX.XX:9092 In the EC2, started kafka 'confluent start' and created 'mytopic' My

问题 We have a "Kerberized Kafka cluster" running brokers version Apache Kafka 0.11.This cluster is managed by a different team and we dont have any control over this. We are now trying to install Kafka Connect cluster on our own K8S cluster. We were following this compatibility matrix https://docs.confluent.io/current/installation/versions-interoperability.html According to this, we had to stick with Confluent Platform 3.3.3 images for Schema Registry and Kafka Connect pods since the Brokers

问题 I have a topic that will eventually have lots of different schemas on it. For now it just has the one. I've created a connect job via REST like this: { "name":"com.mycompany.sinks.GcsSinkConnector-auth2", "config": { "connector.class": "com.mycompany.sinks.GcsSinkConnector", "topics": "auth.events", "flush.size": 3, "my.setting":"bar", "key.converter":"org.apache.kafka.connect.storage.StringConverter", "key.deserializer":"org.apache.kafka.common.serialization.StringDerserializer", "value

问题 I have data in a Kafka topic which I want to preserve on my data lake. Before worrying about the keys, I was able to save the Avro values in files on the datalake using HdfsSinkConnector. The number of message values in each file was determined by the "flush.size" property of the HdfsSinkConnector. All good. Next I wanted to preserve the keys as well. To do this I used the kafka-connect-transform-archive which wraps the String key and Avro value into a new Avro schema. This works great ...

问题 I setup a confluent s3 sink connect, it stores .avro files in s3. I dump those files, and find out that they are just the message itself, I don't know where can I find the message key, any idea? The config is like: { "name": "s3-sink-test", "config": { "connector.class": "io.confluent.connect.s3.S3SinkConnector", "tasks.max": "1", "topics": "book", "s3.region": "eu-central-1", "s3.bucket.name": "kafka", "s3.part.size": "5242880", "storage.class": "io.confluent.connect.s3.storage.S3Storage",