Storing Firehose transfered files in S3 under custom directory names

Question

We primarily do bulk transfer of incoming click stream data through Kinesis Firehose service. Our system is a multi tenant SaaS platform. The incoming click stream data are stor

Answer 1

You can separate your directories by configuring the S3 Prefix. In the console, this is done during setup when you set the S3 bucket name.

Using the CPI, you set the prefix in the --s3-destination-configuration as shown here:

http://docs.aws.amazon.com/cli/latest/reference/firehose/create-delivery-stream.html

Note however, you can only set one prefix per Firehose Delivery Stream, so if you're passing all of your clickstream data through one Firehose Delivery Stream you will not be able to send the records to different prefixes.

Answer 2

Custom prefixes are now supported.