avro

问题 I have the JSON file & JSON Schema to be parsed into the AVRO Schema. I am little bit confused, do i have to write the manual AVRO schema using the data types defined in AVRO documentation. Or is there any automated method / function / program that can work exactly the same as required ? 回答1: Well, you can try https://github.com/fge/json-schema-avro, but they say it is still not complete. So not sure it will work, though 回答2: avro4s derives schemas at compile time from case classes: import

问题 I have the JSON file & JSON Schema to be parsed into the AVRO Schema. I am little bit confused, do i have to write the manual AVRO schema using the data types defined in AVRO documentation. Or is there any automated method / function / program that can work exactly the same as required ? 回答1: Well, you can try https://github.com/fge/json-schema-avro, but they say it is still not complete. So not sure it will work, though 回答2: avro4s derives schemas at compile time from case classes: import

问题 This question is about architecture and kafka topics migrating. Original problem : schema evolution without backward compatibility. https://docs.confluent.io/current/schema-registry/avro.html I am asking the community to give me an advice or share articles from which I can get inspired and maybe think of a solution to my problem. Maybe there is an architecture or streaming pattern. It is not necessary to give me a language specific solution; just give me a direction into which I can go... My

问题 Is there any tool able to create an AVRO schema from a 'typical' JSON document. For example: { "records":[{"name":"X1","age":2},{"name":"X2","age":4}] } I found http://jsonschema.net/reboot/#/ which generates a ' json-schema ' { "$schema": "http://json-schema.org/draft-04/schema#", "id": "http://jsonschema.net#", "type": "object", "required": false, "properties": { "records": { "id": "#records", "type": "array", "required": false, "items": { "id": "#1", "type": "object", "required": false,

问题 I need to mix "record" type with null type in Schema. "name":"specShape", "type":{ "type":"record", "name":"noSpecShape", "fields":[ { "name":"bpSsc", "type":"null", "default":null, "doc":"SampleValue: null" },... For example, for some datas specShape may be null. So if I set type to "name":"specShape", "type":{ **"type":["record", "null"],** "name":"noSpecShape", "fields":[ { "name":"bpSsc", "type":"null", "default":null, "doc":"SampleValue: null" },... it says No type: {"type":["record",

问题 I need to mix "record" type with null type in Schema. "name":"specShape", "type":{ "type":"record", "name":"noSpecShape", "fields":[ { "name":"bpSsc", "type":"null", "default":null, "doc":"SampleValue: null" },... For example, for some datas specShape may be null. So if I set type to "name":"specShape", "type":{ **"type":["record", "null"],** "name":"noSpecShape", "fields":[ { "name":"bpSsc", "type":"null", "default":null, "doc":"SampleValue: null" },... it says No type: {"type":["record",

问题 I need to mix "record" type with null type in Schema. "name":"specShape", "type":{ "type":"record", "name":"noSpecShape", "fields":[ { "name":"bpSsc", "type":"null", "default":null, "doc":"SampleValue: null" },... For example, for some datas specShape may be null. So if I set type to "name":"specShape", "type":{ **"type":["record", "null"],** "name":"noSpecShape", "fields":[ { "name":"bpSsc", "type":"null", "default":null, "doc":"SampleValue: null" },... it says No type: {"type":["record",

问题 Context: We have a Dataflow job that transforms PubSub messages into Avro GenericRecords and writes them into GCS as ".avro". The transformation between PubSub messages and GenericRecords requires a schema. This schema changes weekly with field additions only. We want to be able to update the fields without updating the Dataflow job. What we did: We took the advice from this post and created a Guava Cache that refreshes the content every minute. The refresh function will pull schema from GCS.

问题 I'm trying to use the spark-avro package as described in Apache Avro Data Source Guide. When I submit the following command: val df = spark.read.format("avro").load("~/foo.avro") I get an error: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util

问题 I have the following AVRO schema, which represents a list of "Customer" objects: { "type" : "record", "name" : "List", "fields" : [ { "name" : "Customer", "type" : { "type" : "array", "items" : { "type" : "record", "name" : "Customer", "fields" : [ { "name" : "Sender", "type" : { "type" : "record", "name" : "SenderInfo", "fields" : [ { "name" : "transmitDate", "type" : "long", "source" : "element transmitDate" }, { "name" : "transmitter", "type" : "string", "source" : "element transmitter" },