spring-xd

spring xd losing messages when processing huge volume

人盡茶涼 提交于 2019-12-02 04:09:28
I am using spring xd My stream looks like below and running tests on 3 node container with 1 admin node with rabbit as transport aws-s3|processor1|http-client|processor2>queue:readyQueue I have created below tap. tap1 aws-s3>s3Queue tap2 processor1>processorQueue1 tap3 http-client>httpQueue I run below scenarios in my tests: Scenario1 : 5 files of 200k =1 Million records concurrency of http-client=70 and processor2=30 I see 900k message s3Queue I see 889k message processorQueue1 I see 886k message httpQueue I see 883k message processorQueue2 Messages are lost everywhere and its random

Processing messages through named:channels with prefetch >= 1

自作多情 提交于 2019-11-30 17:04:52
Good evening everybody I run with SpringXD-1.3.0.RELEASE (configuration: 3 admin and 3 container) and RabbitMQ 3.5.7 (3 nodes) into following scenario: stream create --name fire --definition "time --timeUnit=MILLISECONDS > topic:fire" --deploy stream create --name fireeater_1 --definition "topic:fire > null" stream create --name fireeater_2 --definition "topic:fire > null" stream create --name fireeater_3 --definition "topic:fire > null" stream create --name fireeater_4 --definition "topic:fire > null" job create test123 --definition "timestampfile --directory=/tmp" stream deploy fireeater_1 -