Storing Firehose transfered files in S3 under custom directory names

和自甴很熟 提交于 2019-12-06 00:39:24

问题


We primarily do bulk transfer of incoming click stream data through Kinesis Firehose service. Our system is a multi tenant SaaS platform. The incoming click stream data are stored S3 through Firehose. By default, all the files are stored under directories named per given date-format. I would like to specify the directory path for the data files in Firehose planel \ through API in order to segregate the customer data.

For example, the directory structure that I would like to have in S3 for customers A, B and C :

/A/2017/10/12/

/B/2017/10/12/

/C/2017/10/12/

How can I do it?


回答1:


You can separate your directories by configuring the S3 Prefix. In the console, this is done during setup when you set the S3 bucket name.

Using the CPI, you set the prefix in the --s3-destination-configuration as shown here:

http://docs.aws.amazon.com/cli/latest/reference/firehose/create-delivery-stream.html

Note however, you can only set one prefix per Firehose Delivery Stream, so if you're passing all of your clickstream data through one Firehose Delivery Stream you will not be able to send the records to different prefixes.



来源:https://stackoverflow.com/questions/46803567/storing-firehose-transfered-files-in-s3-under-custom-directory-names

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!