Athena can only see the first JSON record written to Firehose by Kinesis Analytics

为君一笑 提交于 2021-02-19 03:53:25

问题


I am using Kinesis Analytics to read in JSON from Kinesis Firehose. I am successfully filtering out some of the records and writing a subset of the JSON properties to another Firehose.

I wanted to execute an Athena query on the data being written to S3 via the destination Firehose. However, the JSON records written to the files in S3 do not have any newlines. Consequently, when I query the data using Athena, it only returns the first record in each file.

When I write records to the source Firehose, I manually insert a newline between records, but Analytics doesn't seem to do this when writing to the destination.

Is there a way to get Analytics to write out a separator or newline between records, so Athena can see all of the records?

来源:https://stackoverflow.com/questions/46605869/athena-can-only-see-the-first-json-record-written-to-firehose-by-kinesis-analyti

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!