avro

问题 Using Azure IOT, I have my device endpoint pointing to Azure Blob storage. Data is sent to the Blob, but i get non-ascii characters and am told that the file extension is incorrect. Sample Blob data is Objavro.codecnullavro.schemaÐ{"type":"record","name":"Message","namespace":"Microsoft.Azure.Devices","fields":[{"name":"EnqueuedTimeUtc","type":"string"},{"name":"Properties","type":{"type":"map","values":"string"}},{"name":"SystemProperties","type":{"type":"map","values":"string"}},{"name":

问题 Using Azure IOT, I have my device endpoint pointing to Azure Blob storage. Data is sent to the Blob, but i get non-ascii characters and am told that the file extension is incorrect. Sample Blob data is Objavro.codecnullavro.schemaÐ{"type":"record","name":"Message","namespace":"Microsoft.Azure.Devices","fields":[{"name":"EnqueuedTimeUtc","type":"string"},{"name":"Properties","type":{"type":"map","values":"string"}},{"name":"SystemProperties","type":{"type":"map","values":"string"}},{"name":

问题 Trying to use confluent_kafka.AvroConsumer to consume messages from a given time stamp. if flag: # creating a list topic_partitons_to_search = list( map(lambda p: TopicPartition('my_topic2', p, int(time.time())), range(0, 1))) print("Searching for offsets with %s" % topic_partitons_to_search) offsets = c.offsets_for_times(topic_partitons_to_search, timeout=1.0) print("offsets_for_times results: %s" % offsets) for x in offsets: c.seek(x) flag=False console returns this Searching for offsets

问题 Using NiFi 1.7.1 (which uses Java Avro 1.8.1) and in the AvroSchemaRegistry, I'm trying to define a schema which has the fields name and app.name at the top level. According to the Avro docs[1] I would assume that I could just define the fullname like normal "name": "app.name" but I hit the error Illegal character in: app.name . It's true that the name portion of the fullname does not allow dots but according to the docs: "If the name specified contains a dot, then it is assumed to be a

问题 Using NiFi 1.7.1 (which uses Java Avro 1.8.1) and in the AvroSchemaRegistry, I'm trying to define a schema which has the fields name and app.name at the top level. According to the Avro docs[1] I would assume that I could just define the fullname like normal "name": "app.name" but I hit the error Illegal character in: app.name . It's true that the name portion of the fullname does not allow dots but according to the docs: "If the name specified contains a dot, then it is assumed to be a

问题 I'm using avro tools to generate a schema file from an avro file in HDFS and dump it to the Linux file system using this command: hadoop jar /usr/bin/Avro/avro-tools-1.8.1.jar getschema /dw/hpm/ap_drg/ap_drg.avro > usr/bin/StageSchema/ap_drg.avsc This works great and gets me the file I need. However; I would like the schema file to be in HDFS and not the Linux file system. How can I change this command to accomplish this? Is there another way I should be doing this? 回答1: Played around for a

问题 I am writing a simple twitter program, where I am reading Tweets using Kafka and want to use Avro for serialization. So far I have just set up twitter configuration in Scala and now want to read tweets using this config. How do I import the following avro schema as defined in the file tweets.avsc in my program? { "namespace": "tweetavro", "type": "record", "name": "Tweet", "fields": [ {"name": "name", "type": "string"}, {"name": "text", "type": "string"} ] } I followed some examples on web

问题 I am writing an Apache Flink streaming application that deserializes data (Avro format) read off a Kafka bus (more details on here). The data is being deserialized into a Scala case class. I am getting an exception when i run the program and it received the first message from Kafka Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.myorg.quickstart.DeviceData.<init>() at org.apache.flink.runtime

问题 I have a JSON file and an avro schema file, which correctly describes it's structure. I then convert the JSON file with the Avro tools into an avro file, without getting an error, like this: java -jar .\avro-tools-1.7.7.jar fromjson --schema-file .\data.avsc .\data.json > .\data.avro I then convert the generated Avro file back to JSON to verify that I got a valid Avro file like this: java -jar .\avro-tools-1.7.7.jar tojson .\data.avro > .\data.json This throws the error: Exception in thread

问题 I can ssh to our box and do a hadoop fs -ls /theFolder and browse in for the files, etc.. but that's all I know too :) My goal is to copy one of those files - they are Avro - on to my local home folder. How can do this? I found also a get command but not sure how to sue that either. 回答1: First, use hadoop fs -get /theFolder to copy it into the current directory you are ssh'ed into on your box. Then you can use either scp or my preference of rsync to copy the files between your box and your