Azure IOT Apache Avro format

问题

Using Azure IOT, I have my device endpoint pointing to Azure Blob storage. Data is sent to the Blob, but i get non-ascii characters and am told that the file extension is incorrect. Sample Blob data is

Objavro.codecnullavro.schemaÐ{"type":"record","name":"Message","namespace":"Microsoft.Azure.Devices","fields":[{"name":"EnqueuedTimeUtc","type":"string"},{"name":"Properties","type":{"type":"map","values":"string"}},{"name":"SystemProperties","type":{"type":"map","values":"string"}},{"name":"Body","type":["null","bytes"]}]} ±¡RëZË8è€‘,¡…$• à82018-02-20T15:12:16.1060000Z $connectionDeviceIdLane4Free(connectionAuthMethodœ{"scope":"device","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}8connectionDeviceGenerationId$636543931333384343enqueuedTime82018-02-20T15:12:16.1060000Z d{"deviceId": "myPythonDevice","PinsCCC": 302.00,\}82018-02-20T15:12:16.2000000Z

Azure IOT docs says that the format is Apache Avro, but I can't find anything on how to decode the non-ascii characters. I don't see where to specify a json extension.

回答1:

It seems by designed.

You can utilize Avro tool to read or convert to JSON format.

Here is a Python sample to read Avro file.

回答2:

From you original question, seems you need to read the content from ARVO file. Following are the steps to use C# to do that from Azure storage:

Install Microsoft.Avro.Core nugget packege

Add following namespaces:

using Microsoft.Hadoop.Avro;
using Microsoft.Hadoop.Avro.Container;

Get AVRO records

private async Task<List<AvroRecord>> GetAvroRecordsAsync(CloudBlockBlob cloudBlockBlob)
{
    var memoryStream = new MemoryStream();
    await cloudBlockBlob.DownloadToStreamAsync(memoryStream);
    memoryStream.Seek(0, SeekOrigin.Begin);
    List<AvroRecord> avroRecords;
    using (var reader = AvroContainer.CreateGenericReader(memoryStream))
    {
        using (var sequentialReader = new SequentialReader<object>(reader))
        {
              avroRecords = sequentialReader.Objects.OfType<AvroRecord>().ToList();
        }
    }

    return avroRecords;
}

Deserialize the object from avro record:

private MyObject GetMyObject(AvroRecord avroRecord)
{
    var body = avroRecord.GetField<byte[]>("Body");
    var dataBody= Encoding.UTF8.GetString(body);
    var myObj= JsonConvert.DeserializeObject<MyObject>(dataBody);
    return myObj;
}

Hope this will be helpful to you.

来源：https://stackoverflow.com/questions/49013821/azure-iot-apache-avro-format

标签

azure

iot

avro