amazon-kinesis-firehose

Before sending the data I am using JSON.stringify to the data and it looks like this {"data": [{"key1": value1, "key2": value2}, {"key1": value1, "key2": value2}]} But once it passes through AWS API Gateway and Kinesis Firehose puts it to S3 it looks like this { "key1": value1, "key2": value2 }{ "key1": value1, "key2": value2 } The seperator comma between the JSON objects are gone but I need it to process data properly. Template in the API Gateway: #set($root = $input.path('$')) { "DeliveryStreamName": "some-delivery-stream", "Records": [ #foreach($r in $root.data) #set($data = "{ ""key1"": ""

I have a Firehose stream that is intended to ingest millions of events from different sources and of different event-types. The stream should deliver all data to one S3 bucket as a store of raw\unaltered data. I was thinking of partitioning this data in S3 based on metadata embedded within the event message like event-souce, event-type and event-date. However, Firehose follows its default partitioning based on record arrival time. Is it possible to customize this partitioning behavior to fit my needs? No. You cannot 'partition' based upon event content. Some options are: Send to separate

Consider the following: A table in Redshift called 'people' that has fields id, name and age A kinesis firehose stream called 'people' that is configured to write to the 'people' table and the value for 'Redshift table columns' is 'id,name,age' It's not clear how to format the 'Data' blob. Here's an example of what the code looks like with the data separated by tabs: let AWS = require('aws-sdk'); let firehose = new AWS.Firehose(); let params = { DeliveryStreamName: 'people', // id,name,age Records: [{Data: '4ccf6d3a-acdf-11e5-ad54-28cfe91fa8f1\tBob\tSmith'}] }; firehose.putRecordBatch(params,

We primarily do bulk transfer of incoming click stream data through Kinesis Firehose service. Our system is a multi tenant SaaS platform. The incoming click stream data are stored S3 through Firehose. By default, all the files are stored under directories named per given date-format. I would like to specify the directory path for the data files in Firehose planel \ through API in order to segregate the customer data. For example, the directory structure that I would like to have in S3 for customers A, B and C : / A /2017/10/12/ / B /2017/10/12/ / C /2017/10/12/ How can I do it? You can

How can we invoke multiple AWS Lambda functions one after the other ? For example if an AWS Lambda chain consists of 8 separate lambda functions and each simulate a 1 sec processing event and then invoke the next function in the chain. I wouldn't recommend using direct invoke to launch your functions. Instead you should consider creating an SNS Topic and subscribing your Lambda functions to this topic. Once a message is published to your topic, all functions will fire at the same time. This solution is also easily scalable. See more information at official documentation Invoking Lambda

The tool below is a batch import method of copying data from SQL Server RDS into Redshift. AWS Schema Conversion Tool Exports from SQL Server to Amazon Redshift Is there a more streamlined method, conducting every second way of streaming data from MS SQL Server into Redshift with Kinesis Firehose. I know we can move AWS Aurora SQL directly into Redshift with Kinesis. If your goal is to move data from Microsoft SQL Server into Amazon Redshift, then you could consider using AWS Database Migration Service . It can copy data as a one-off job but can also migrate on a continuing basis. See: Using a