confluent

问题 I am using the confluent to import data from kafka to hive, trying to do the same thing as this: Bucket records based on time(kafka-hdfs-connector) my sink config is like this: { "name":"yangfeiran_hive_sink_9", "config":{ "connector.class":"io.confluent.connect.hdfs.HdfsSinkConnector", "topics":"peoplet_people_1000", "name":"yangfeiran_hive_sink_9", "tasks.max":"1", "hdfs.url":"hdfs://master:8020", "flush.size":"3", "partitioner.class":"io.confluent.connect.hdfs.partitioner

问题 Can AVRO schemas be registered with confluent schema registry service ? As per readme on github https://github.com/confluentinc/schema-registry Every example uses a JSON schema with a single field and type without any name. I am trying to store following schema to repository but with different variants getting different error. curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{"type": "record","name": "myrecord","fields": [{"name": "serialization",

问题 I have a Kafka Avro Topic generated using KafkaAvroSerializer. My standalone properties are as below. I am using Confluent 4.0.0 to run Kafka connect. key.converter=io.confluent.connect.avro.AvroConverter value.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=<schema_registry_hostname>:8081 value.converter.schema.registry.url=<schema_registry_hostname>:8081 key.converter.schemas.enable=true value.converter.schemas.enable=true internal.key.converter=org

问题 I am running into a strange behaviour with my Kafka producer and consumer. Below is my setup on my local machine 1 zookeper node 2 kafka broker nodes 1 producer (doing async writes) and 1 subscriber written in go using this library I am creating a topic using kafka's command line tool as below ./kafka-topics.sh --zookeeper localhost:2181 --create --topic foo --partitions 1 --replication-factor 2 --config min.insync.replicas=2 The issue is that whenever i kill leader node of the partition, the

问题 I have created a source and a sink connector for kafka connect Confluent 5.0, to push two sqlserver tables to my datalake Here is my SQLServer table schema : CREATE TABLE MYBASE.dbo.TABLE1 ( id_field int IDENTITY(1,1) NOT NULL, my_numericfield numeric(24,6) NULL, time_field smalldatetime NULL, CONSTRAINT PK_CBMARQ_F_COMPTEGA PRIMARY KEY (id_field) ) GO My Cassandra schema : create table TEST-TABLE1(my_numericfield decimal, id_field int, time_field timestamp, PRIMARY KEY (id_field)); Here is

问题 I've a kafka topic and each message in the topic has lat/lon and event timestamp. Created a stream referring to topic and would like to calculate distance between 2 point using geo_distance . example GpsDateTime lat lon 2016-11-30 22:38:36, 32.685757, -96.735942 2016-11-30 22:39:07, 32.687347, -96.732841 2016-11-30 22:39:37, 32.68805, -96.729726 I would like to create a new stream on the above stream and enrich it with distance. GpsDateTime lat lon Distance 2016-11-30 22:38:36, 32.685757, -96

问题 I am trying to make use of mirror maker to replicate __consumre_offsets topic along with other topics. It is giving error as mentioned below. [2018-10-24 16:16:03,802] ERROR Error when sending message to topic __consumer_offsets with key: 16 bytes, value: 445 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. [2018-10-24 16:16:03,828] ERROR

问题 I am using kafka stream download from Confluent (http://www.confluent.io/product/kafka-streams/). I am following the instructions to run Zookeeper and Kafka on Windows. But while I try to start ZooKeeper using the command D:\Softwares\confluent-3.0.1\bin\windows>zookeeper-server-start.bat ./etc/kafka/zookeeper.properties , I get the error D:\Softwares\confluent-3.0.1\bin\windows../../etc/kafka/log4j.properties was unexpected at this time. If I check the "zookeeper-server-start.bat" file the

问题 I am using confluent for to connect my DB and ES getting exception as: org.apache.kafka.connect.errors.DataException: STRUCT is not supported as the document id. at io.confluent.connect.elasticsearch.DataConverter.convertKey(DataConverter.java:75) at io.confluent.connect.elasticsearch.DataConverter.convertRecord(DataConverter.java:84) at io.confluent.connect.elasticsearch.ElasticsearchWriter.write(ElasticsearchWriter.java:210) at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.put

问题 I am curious about the advantages and disadvantages of Confluent's Kafka REST Proxy and the producer/consumer implemented with the kafka official client library. i know that Confluent's Kafka REST Proxy is used for administrative tasks and for languages ​​not supported by the kafka client. So, what are the advantages of the kafka client? 回答1: One advantage of a native client would be raw performance via direct TCP to the brokers rather than round trip HTTP serialization + JVM serialization