How to poll from a queue 1 message at a time after downstream flow is completed in Spring Integration

问题

I am currently working on improving performance in an integration flow trying to parallelize message processing. I have implemented all using Java DSL.

The current Integration flow takes messages from a Queue Channel with a fixed Poller and process the message serially through multiple handlers one after the other until it reaches a final handler that makes some final calculations taking into account each of the previous handler's output. They are all wired up within the same Integration Flow. And basically those handlers wrap calls to external systems. The important thing that I need to preserve here is that a message must not be taken from the queue until all the downstream flow is completed for the previous one. What I need to parallelize are the handlers.

Current Integration Flow: MessageQueue -> Poller -> Handler 1 -> Handler 2 -> Handler X -> Final Handler

I tried to incorporate parallelism doing the following and it works pretty well.

MessageQueue -> Poller -> Splitter -> Executor -> Router with subflow mappings to the different Handlers -> Aggregator -> Final Handler

The problem I found with this approach is that a new message is taken from the Queue Channel before the previous one passes through all the downstream flow. It is pretty clear why, adding the Splitter and Executor change the way messages are attended but the thing is there could be a dependency between the outcome of to different messages.

The question is, how can I retrieve messages from the Queue Channel one at a time like kind of "suspend" the Poller until the message being processed goes down to the last endpoint after the Aggregator? I don't know how to rearrange the components or what else I can do to achieve that.

Sorry I tried to look for the answer but I couldn't find it... need some guidance here please. Thanks so much

@Blink this is what worked for me, probably needs some refactor and I am sure it could be written more elegantly. I am not an expert, sorry.

Well the basic elements are:

an Interface to wrap the Messaging System

a Message Channel where the Message will be routed when calling the gateway method

@Bean
public DirectChannel integrationChannel() {
    return MessageChannels.direct().get();
}

@MessagingGateway
interface WrappingGateway {

    @Gateway(requestChannel = "integrationChannel")
    TrackingLog executeIntegration(TrackingLog trackingLog);

}

TrackingLog is a model that I use to register outcomes along the downstream flow.

And basically I call the wrapping Gateway within the Integration Flow that pulls messages from the Message Queue.

@Autowired
WrappingGateway integrationGateway;

@Bean
public IntegrationFlow createCatalogueChannelFlow() {
    return IntegrationFlows.from(cataloguePriorityChannel())

            // Queue Poller
            .bridge(s -> s.poller(Pollers.fixedRate(1, TimeUnit.SECONDS).maxMessagesPerPoll(1)).autoStartup(true)
                    .id("cataloguePriorityChannelBridge"))

            // Call to Gateway 
            .handle(m -> {
                this.integrationGateway
                        .executeIntegration(((TrackingLog) m.getPayload()));
            })

            .get();
}

@Bean
public IntegrationFlow startCatalogueIntegrationChannelFlow() {
    return IntegrationFlows.from(integrationChannel())

            // Log
            .handle(trackerSupportClient, "logMessagePreExecution")

            // Set TrackingLog in message Header
            .enrichHeaders(e -> e.headerFunction("TRACKING_LOG", m -> {
                return ((TrackingLog) m.getPayload());
            }))
 ....

The whole integration is a little bit more complex, it starts with an Async HTTP Gateway, transformers, routers, stores in mongodb, etc. The point here is that as @Artem Bilan suggested me, the call to the gateway blocks the thread and prevents the Queue Poller from getting more messages until the current one is completely processed.

Hope this helps you.

回答1:

That's indeed an interesting task... I'll share with you my thoughts and you'll choose what best fits for you.

We always can wrap a part of flow into a @MessagingGateway which should wait for reply. And it is already doesn't matter how async is its sub-flow. So, you can perform those your tasks in parallel, but gateway would still wait for reply in the main thread blocking the next poll from the queue. You should ensure that you return something in the end of sub-flow into the replyChannel to unblock the main thread. See docs here: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/messaging-endpoints.html#gateway
We have a BarrierMessageHandler component out-of-the-box. It's point is indeed to block the current thread with a message until some trigger arrives for the correlation the message belongs. Only the problem with this component that you need to figure out how to release a barrier for the first message since exactly this message is going to be as a trigger for the next one. Although we probably can use a one-time-use router to bypass that barrier for the first message... Docs are here: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/message-routing.html#barrier
We have a component like MessageSourcePollingTemplate. So, you can call the QueueChannel wrapped into a MessageSource lambda whenever you need to do that. I kinda can't come up right now how that could fit into a flow, but this one more idea how to suspend polling. See docs: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/core.html#deferred-acks-message-source
Another way is to add a MethodInterceptor into a Poller configuration to skip calling invocation.proceed() if some AtomicBoolean for state is true. This way you keep a state until the message is processed and every poll task is going to skip until you reset that state. Docs: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/messaging-endpoints.html#endpoint-pollingconsumer

来源：https://stackoverflow.com/questions/61355111/how-to-poll-from-a-queue-1-message-at-a-time-after-downstream-flow-is-completed

标签

parallel-processing

spring-integration