spring-integration-aws

Spring Aws Kinesis Binder Acquiring and Releasing lock issues in Dynamo DB while consuming messages

蓝咒 提交于 2021-02-10 16:18:49
问题 Sometimes, when we are stopping the application abruptly exception occurs that unlocking has been failed. Then the same group will never gets the messages. Other groups are getting messages. I am using the aws kinesis binder snapshot version. This is the error, when application is stopped. 2018-07-19 22:21:21.371 ERROR 60981 --- [s-shard-locks-1] a.i.k.KinesisMessageDrivenChannelAdapter : Error during unlocking: DynamoDbLock [lockKey=aaaa:myStream:shardId-000000000000,lockedAt=2018-07-19@22

Spring cloud stream Special Chars in Message received from kinesis

杀马特。学长 韩版系。学妹 提交于 2020-01-04 09:25:00
问题 When I consume the message from kinesis stream. I get some junk chars with headers etc @StreamListener(Processor.INPUT) public void receive(String message) { System.out.println("Message recieved: "+message); throw new RuntimeException("Exception thrown"); } @StreamListener("errorChannel") public void transform(ErrorMessage errorMessage) throws UnsupportedEncodingException { //original paylaod System.out.println("Error Oiginal Message Payload"+new String((byte[])errorMessage.getOriginalMessage

S3: Outbound adapter to place file in multiple target buckets

假如想象 提交于 2019-12-31 05:10:47
问题 have a spring boot application, where I am trying to place a file into multiple S3 bucket using single S3 outbound adapter.. Would like to know if its possible to place the file in multiple bucket using single outbound adapter using spring-integration-aws itself( without using aws -sdk) Any suggestion will be helpful. S3 : Outbound adapter: <int-aws:s3-outbound-channel-adapter id="filesS3Mover" channel="filesS3MoverChannel" transfer-manager="transferManager" bucket="${aws.s3.target.bucket}"

S3: Outbound adapter to place file in multiple target buckets

早过忘川 提交于 2019-12-31 05:10:08
问题 have a spring boot application, where I am trying to place a file into multiple S3 bucket using single S3 outbound adapter.. Would like to know if its possible to place the file in multiple bucket using single outbound adapter using spring-integration-aws itself( without using aws -sdk) Any suggestion will be helpful. S3 : Outbound adapter: <int-aws:s3-outbound-channel-adapter id="filesS3Mover" channel="filesS3MoverChannel" transfer-manager="transferManager" bucket="${aws.s3.target.bucket}"

How to check if aws s3 bucket available before uploading file using outbound gateway

孤街浪徒 提交于 2019-12-24 11:36:31
问题 using aws s3 outbound adapter in spring boot application,trying to upload files in s3 bucket. would like to check if bucket available before uploading file. if bucket is not available need to throw error. suggest on this. <int-aws:s3-outbound-gateway id="FileChannelS3" request-channel="filesOutS3CChainChannel" reply-channel="filesArcChannel" transfer-manager="transferManager" bucket-expression="headers.TARGET_BUCKET" command="UPLOAD"> <int-aws:request-handler-advice-chain> <ref bean=

Unable to consume messages as batch mode in Kinesis Binder

我是研究僧i 提交于 2019-12-13 18:47:03
问题 I am trying to consume messages from Kinesis stream as batch I am using compile('org.springframework.cloud:spring-cloud-starter-stream-kinesis:1.0.0.BUILD-SNAPSHOT') Application.yml spring: cloud: stream: bindings: input: group: groupName destination: stream-name content-type: application/json consumer: listenerMode: batch idleBetweenPolls: 10000 Code As per the documentation, when listenerMode is batch, then it is expected to have list as payload @StreamListener(Sink.INPUT) public void

spring-integration-aws dynamic file download

旧巷老猫 提交于 2019-12-13 03:43:22
问题 I've a requirement to download a file from S3 based on a message content. In other words, the file to download is previously unknown, I've to search and find it at runtime. S3StreamingMessageSource doesn't seem to be a good fit because: It relies on polling where as I need to wait for the message. I can't find any way to create a S3StreamingMessageSource dynamically in the middle of a flow. gateway(IntegrationFlow) looks interesting but what I need is a gateway(Function<Message<?>,

Timeout waiting for connection from pool while polling S3 for Objects

别说谁变了你拦得住时间么 提交于 2019-12-11 18:45:15
问题 I am working on a backend service which polls S3 bucket periodically using spring aws integration and processes the polled object from S3. Below is the implementation for it @Configuration @EnableIntegration @IntegrationComponentScan @EnableAsync public class S3PollerConfiguration { //private static final Logger log = (Logger) LoggerFactory.getLogger(S3PollerConfiguration.class); @Value("${amazonProperties.bucketName}") private String bucketName; @Bean @InboundChannelAdapter(value =

IdleBetween pools not pulling Messages as specified

故事扮演 提交于 2019-12-11 18:33:12
问题 I am consuming messages in batch mode. I wanted to pull 8 messages each 250 ms from stream. spring: cloud: stream: kinesis: bindings: input: consumer: listenerMode: batch idleBetweenPolls: 250 recordsLimit: 8 bindings: input: group: my-group destination: stream content-type: application/json I have pushed around 100 messages on to stream, and I started the consumer. As per configuration, I am supposed to pull messages each 250 ms. But the poller not pulling messages each 250 ms.

Spring Integration: MessageSource doesn't honor errorChannel header

扶醉桌前 提交于 2019-12-11 14:10:09
问题 I've the following flow: @Resource(name = S3_CLIENT_BEAN) private MessageSource<InputStream> messageSource; public IntegrationFlow fileStreamingFlow() { return IntegrationFlows.from(s3Properties.getFileStreamingInputChannel()) .enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true)) .handle(String.class, (fileName, h) -> { if (messageSource instanceof S3StreamingMessageSource) { S3StreamingMessageSource s3StreamingMessageSource = (S3StreamingMessageSource) messageSource;