Does Amazon Kinesis Firehose support Data Transformations programatically?

删除回忆录丶 提交于 2020-06-03 09:55:53

问题


I have a use case in which I have to verify that the payloads sent to Kinesis firehose are indeed being sent.

In order to do that I came up with the chain Firehose -> Firehose Data transformation(using lambda) -> DDB -> Check for payload in DDB (the payload is the hashkey in the DDB). I have to define this entire chain in one shot programatically. The data transformation is the same as http://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html.

I am doing all this since I cannot completely control the file name in the S3 bucket it goes to. So I need the exact payload being sent into some persistent key value store.

The problem is

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesisfirehose/model/CreateDeliveryStreamRequest.html does not seem to support adding a data transformation lambda.

My question is, is this doable without touching the console even once (completely through the AWS Kinesis Firehose API).

Or does have any alternate suggestions to move the data to DDB somehow.


回答1:


Well so I figured it out after much effort and documentation scrounging.

You would have to define a processing configuration with a lambda ARN to define a data transform.

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesisfirehose/model/ProcessingConfiguration.html

Here is what such a configuration would look like in code.

final ProcessingConfiguration processingConfiguration =
      new ProcessingConfiguration().withEnabled(true)
         .withProcessors(newProcessor().withType(ProcessorType.Lambda)
         .withParameters(new ProcessorParameter().withParameterName(LambdaArn)
         .withParameterValue(lamdbaFunctionArn)));

final CreateDeliveryStreamResult describeDeliveryStreamResult =
      client.createDeliveryStream(new CreateDeliveryStreamRequest().withExtendedS3DestinationConfiguration(
         new ExtendedS3DestinationConfiguration()
             .withBucketARN(s3BucketARN)
             .withRoleARN(roleArn)
             .withPrefix(keyPrefix)
             .withProcessingConfiguration(processingConfiguration))
             .withDeliveryStreamName(streamName));

Here the ARNs are resource names for various objects (S3 destination, data transform lambda, IAM role etc).




回答2:


You can have a lambda function to specify the task. https://github.com/hixichen/golang_lamda_decode_protobuf_firehose



来源:https://stackoverflow.com/questions/43665829/does-amazon-kinesis-firehose-support-data-transformations-programatically

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!