confluent-schema-registry

问题 I'm using Apache Beam's kafkaIO to read from a topic that has an avro schema in Confluent schema registry. I'm able to deserialize the message and write to files. But ultimately i want to write to BigQuery. My pipeline isn't able to infer the schema. How do I extract/infer the schema and attach it to the data in the pipeline so that my downstream processes (write to BigQuery) can infer the schema? Here is the code where I use the schema registry url to set the deserializer and where i read

问题 I want to read data from a confluent cloud topic and then write in another topic. At localhost, I haven't had any major problems. But the schema registry of confluent cloud requires to pass some authentication data that I don't know how to enter them: basic.auth.credentials.source=USER_INFO schema.registry.basic.auth.user.info=: schema.registry.url=https://xxxxxxxxxx.confluent.cloudBlockquote Below is the current code: import com.databricks.spark.avro.SchemaConverters import io.confluent

问题 I want to read data from a confluent cloud topic and then write in another topic. At localhost, I haven't had any major problems. But the schema registry of confluent cloud requires to pass some authentication data that I don't know how to enter them: basic.auth.credentials.source=USER_INFO schema.registry.basic.auth.user.info=: schema.registry.url=https://xxxxxxxxxx.confluent.cloudBlockquote Below is the current code: import com.databricks.spark.avro.SchemaConverters import io.confluent

问题 I am using Kafka Connector in Apache Flink for access to streams served by Confluent Kafka . Apart from schema registry url ConfluentRegistryAvroDeserializationSchema.forGeneric(...) expecting 'reader' schema. Instead of providing read schema I want to use same writer's schema(lookup in registry) for reading the message too because Consumer will not have latest schema. FlinkKafkaConsumer010<GenericRecord> myConsumer = new FlinkKafkaConsumer010<>("topic-name",

问题 I have aws MSK set up and i am trying to sink records from MSK to elastic search. I am able to push data into MSK into json format . I want to sink to elastic search . I am able to do all set up correctly . This is what i have done on EC2 instance wget /usr/local http://packages.confluent.io/archive/3.1/confluent-oss-3.1.2-2.11.tar.gz -P ~/Downloads/ tar -zxvf ~/Downloads/confluent-oss-3.1.2-2.11.tar.gz -C ~/Downloads/ sudo mv ~/Downloads/confluent-3.1.2 /usr/local/confluent /usr/local

问题 I have aws MSK set up and i am trying to sink records from MSK to elastic search. I am able to push data into MSK into json format . I want to sink to elastic search . I am able to do all set up correctly . This is what i have done on EC2 instance wget /usr/local http://packages.confluent.io/archive/3.1/confluent-oss-3.1.2-2.11.tar.gz -P ~/Downloads/ tar -zxvf ~/Downloads/confluent-oss-3.1.2-2.11.tar.gz -C ~/Downloads/ sudo mv ~/Downloads/confluent-3.1.2 /usr/local/confluent /usr/local

问题 I spinned on my laptop a Kafka in Docker (with docker-compose). After that, created new kafka topic with: kafka-topics --zookeeper localhost:2181 --create --topic simple --replication-factor 1 --partitions 1 (did not create schema in Schema Registry yet). Now trying to produce (based on this example - step 3 - https://docs.confluent.io/4.0.0/quickstart.html): kafka-avro-console-producer \ --broker-list localhost:9092 --topic simple \ --property value.schema='{"type":"record","name":"myrecord"

问题 I have a use case where I have a JSON and I want to generate schema and record out of the JSON and publish a record. I have configured the value serializer and Schema setting is Backward compatible. First JSON String json = "{\n" + " \"id\": 1,\n" + " \"name\": \"Headphones\",\n" + " \"price\": 1250.0,\n" + " \"tags\": [\"home\", \"green\"]\n" + "}\n" ; Version 1 schema registered. Received message in avro console consumer. Second JSON. String json = "{\n" + " \"id\": 1,\n" + " \"price\":

问题 I have a use case where I have a JSON and I want to generate schema and record out of the JSON and publish a record. I have configured the value serializer and Schema setting is Backward compatible. First JSON String json = "{\n" + " \"id\": 1,\n" + " \"name\": \"Headphones\",\n" + " \"price\": 1250.0,\n" + " \"tags\": [\"home\", \"green\"]\n" + "}\n" ; Version 1 schema registered. Received message in avro console consumer. Second JSON. String json = "{\n" + " \"id\": 1,\n" + " \"price\":

问题 Can AVRO schemas be registered with confluent schema registry service ? As per readme on github https://github.com/confluentinc/schema-registry Every example uses a JSON schema with a single field and type without any name. I am trying to store following schema to repository but with different variants getting different error. curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{"type": "record","name": "myrecord","fields": [{"name": "serialization",