spring-xd

SpringXD and Spring Integration: Read from kafka topic every X minutes, then send to another Topic

青春壹個敷衍的年華 提交于 2019-12-08 09:35:21
问题 I'm trying to implement a solution to create a SpringXD stream composed by a kafka source, a bridge module, and a kafka sink. So I have something like: <channel id="pollable"> <queue /> </channel> <bridge input-channel="pollable" output-channel="executorChannel"> <poller max-messages-per-poll="5" fixed-rate="5000" /> </bridge> My problem is that I would like to avoid the poller somehow. Basically because I would like to avoid keeping messages in memory when those messages are in the queue. I

creating a custom websocket sink for spring-xd

浪尽此生 提交于 2019-12-08 07:56:18
问题 I am pretty new to spring-xd and need some advice for creating a custom sink. Specifically, I would like to create a sink that registers a Websocket. The general idea is that module upload --file "websocket-sink-0.0.1-SNAPSHOT.jar" --type "sink" --name "websocket-sink" stream create --name "websocket-sink-test" --definition "http --port=9191 | websocket-sink --port=9292" --deploy should install the sink module and create a stream that accepts http input on port 9191 and sends the payload to

Spring Batch - Read files from Aws S3

给你一囗甜甜゛ 提交于 2019-12-07 10:39:15
问题 I am trying to read files from AWS S3 and process it with Spring Batch: Can a Spring Itemreader process this Task? If so, How do I pass the credentials to S3 client and config my spring xml to read a file or multiple files <bean id="itemReader" class=""org.springframework.batch.item.file.FlatFileItemReader""> <property name="resource" value=""${aws.file.name}"" /> </bean> 回答1: Update To use the Spring-cloud-AWS you would still use the FlatFileItemReader but now you don't need to make a custom

creating a custom websocket sink for spring-xd

独自空忆成欢 提交于 2019-12-06 16:19:21
I am pretty new to spring-xd and need some advice for creating a custom sink. Specifically, I would like to create a sink that registers a Websocket. The general idea is that module upload --file "websocket-sink-0.0.1-SNAPSHOT.jar" --type "sink" --name "websocket-sink" stream create --name "websocket-sink-test" --definition "http --port=9191 | websocket-sink --port=9292" --deploy should install the sink module and create a stream that accepts http input on port 9191 and sends the payload to the websocket sink where clients can connect (via port 9292 ) and consume that data. I pretty much

How to implement a Spring XD sink?

五迷三道 提交于 2019-12-06 11:24:22
So far I have implemented Spring XD processors, e.g. like this: @MessageEndpoint public class MyTransformer { @Transformer( inputChannel = "input", outputChannel = "output" ) public String transform( String payload ) { ... } }; However, I am stuck at implementing a custom sink now. The current documentation is not very helpful, since it simply configures something "magically" via XML: <beans ...> <int:channel id="input" /> <int-redis:store-outbound-channel-adapter id="redisListAdapter" collection-type="LIST" channel="input" key="${collection}" auto-startup="false"/> <beans:bean id=

Kafka Source in Spring Cloud Data Flow

ε祈祈猫儿з 提交于 2019-12-05 02:40:27
问题 I am migrating from Spring XD to Spring Cloud Data Flow. When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow - One of them is KAFKA source. My question is why KAFKA source is removed from standard sources list in spring cloud data flow ? 回答1: When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow Majority of the applications are ported over and the remaining are incrementally prioritized -

Future of Spring-XD [closed]

爱⌒轻易说出口 提交于 2019-12-04 07:14:33
问题 Closed . This question needs details or clarity. It is not currently accepting answers. Want to improve this question? Add details and clarify the problem by editing this post. Closed 3 years ago . I heared rumors that Spring XD lost development drive and focus at Spring shifts towards cloud services. Since I'm considering XD (the otehr one I consider is Apache Beam, though not yet mature enough imho), I'd like to get some facts / release plan for the future of XD instead of relying on rumors

How to get Spring XD to deploy a predefined set of streams and taps on startup

本秂侑毒 提交于 2019-12-03 14:51:08
Is there a way to launch a set of steams and taps that have been defined when a new instance of spring xd is launched? The scenario is this: We've done some prototyping on our dev systems, but now when we'd like to build and release this to a test team. We'd like to automate most of the background work and the user needs to just worry about the output from the streams and not have to worry about the streams or their definitions or their deployment. To facilitate this is there any runtime configurations we can use? UPDATE: Our current approach involves writing a shell script to make some rest

Future of Spring-XD [closed]

强颜欢笑 提交于 2019-12-02 15:04:51
I heared rumors that Spring XD lost development drive and focus at Spring shifts towards cloud services. Since I'm considering XD (the otehr one I consider is Apache Beam, though not yet mature enough imho), I'd like to get some facts / release plan for the future of XD instead of relying on rumors. The Spring XD homepage has an announcement at the bottom of the page: Announcement Building upon Spring Boot and Spring Cloud capabilities, Spring XD is actively being redesigned as Spring Cloud Data Flow - a cloud native programming and operating model for composable message-driven microservices.

spring xd losing messages when processing huge volume

谁说我不能喝 提交于 2019-12-02 11:11:04
问题 I am using spring xd My stream looks like below and running tests on 3 node container with 1 admin node with rabbit as transport aws-s3|processor1|http-client|processor2>queue:readyQueue I have created below tap. tap1 aws-s3>s3Queue tap2 processor1>processorQueue1 tap3 http-client>httpQueue I run below scenarios in my tests: Scenario1 : 5 files of 200k =1 Million records concurrency of http-client=70 and processor2=30 I see 900k message s3Queue I see 889k message processorQueue1 I see 886k