Service Fabric Reliable Services Pipeline design

Question

I need to implement pipeline if Service Fabric's Reliable Services, and I need some guidelines about what of these approaches is preferable from the viewpoint of reliability simplicity and simple good design:

Answer 1

I have been investigating this topic a lot as well (to be applied to my work for NServiceBus and MessageHandler) and would like to provide my thoughts on the matter. However I haven't determined what the best model is yet.

If you disregard the practical implementation with ServiceFabric I would categorize the proposed approach in the following order when it comes to reliability:

• C) The store and forward model is probably the best of the 3 models when it comes to interservice communication, all services can work independently from each other and are in no way subject to networking outages (at the downside of added latency)
• A) Input queue per service: Each service free from impact by network outages for it's own work. However when it wishes to send messages to another service it may be impacted by network outages and needs retry built in to accomodate for this.
• B) Output queue per service: Is probably the least of the 3 models as each service is directly dependent on a resource of the others, this results in to much dependency on network availability between the nodes.

If you look at it from a simplicity point of view, I would categorize them the following way

• A) Input queue per service: As the message source needs to actively route messages to a given destination queue, it is fairly simpel to implement business processes or workflows (what I assume your pipeline is going to represent) using a routing pattern (either static routing or dynamic f.e. using a routing slip pattern
• C) Store and forward: Again routing is an explicit part of your implementation, so both static and dynamic routing patterns are possible, however the practical implemenation is harder as you need to build and manage a messagepump that transfers messages from the transfer queue (output) to the destination queue and the associated need to flow context from the message source into the message pump. (Shameless plug: NServiceBus is a framework that can take away the complexity for you and make this scenario as simple as A)
• B) Output queue per service: Each service needs to be setup to explicitly read from another's queue, this approach would only allow static routing as the routing rules are embedded in where you read from only (this severely limit you from a functional perspective)

If we take ServiceFabric's implementation details into account, then I assume you want to make use of the IReliableQueue implementation? This implementation has some shortcomings though, that make me wonder if these patterns can actually be implemented properly on ServiceFabric's native storage infrastructure.

1. The storage infrastructure is only available on Statefull services, so Stateless services (like Rest API's or other protocol termination gateway's) cannot be part of the pipeline (usually you want one of these as an entry point)
2. Only 1 thread can access a reliable queue at the same time, so it is impossible to write and read from the same queue at the same time. This severely limits throughput of the queue.
3. Accessing a reliable queue requires a local transaction, but these transactions are limited to a single partition. So it's also impossible to scale out your statefull services to create a competing consumer pattern.

Given these shortcomings, I'm still inclined to use another type of queueing infrastructure for SF Services instead of SF's persistence model, for example Azure Service Bus or Azure Storage Queues (Which NserviceBus allows as well).

In short, I'll support both A and C, with a slight preference for C, but I'm not convinced about using reliable queues as an implementation until these shortcomings have been resolved.