问题
My function is sending a payload to different sftp servers. Those servers are limited in how many connections they can accept.
I need a solution to throttle our connections to those servers.
The function is triggered by storage queues, and the first draft of the design is:
I then learned that you can only have 1 trigger per function, which led me to sandwhich another aggregating queue:
I can set the batchSize/newBatchThreshold on the originating queues, but I'm not certain this will work because the originating queues will not be aware of when to push messages to the aggregate queue.
- I need the function to not scale out to more than N instances for all messages from queue X, since the sftp server X will not accept more than N connections.
- Additionally, I need the function to scale out to no more than M instances for all messages from queue Y, since the sftp server Y will not accept more than M connections.
The total instances would be M + N for the above scenario.
How do we adjust our design in order to fit these requirements?
回答1:
There's no 100% bullet-proof solution to this, the issue is tracked here.
The best way could be setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 in application settings of the Function App which is triggered by the aggregate queue. Then, you should only get one concurrent instance of the Function App, so the batchSize setting will actually be useful for rate limiting.
You don't need to limit queue processors X/Y/Z in this case, let the messages flow to the aggregate.
Now, I didn't understand if only messages from queue X touch SFPT X, or it's many-to-many. If it's one-on-one, it makes sense to get rid of the aggregate queue, have three Functions and limit the concurrency for each of the queues separately.
Anyway, the limit settings are as I suggested above.
In case this still doesn't satisfy your requirements, you may switch to another messaging service. For instance, send all messages of one type into a separate session of Service Bus or a single partition of Event Hub, which will naturally limit the concurrency on the broker level.
回答2:
Option 1: Depend on the sftp's error response
Does the sftp server return a 429 (too many requests) response? Or something similar? When you get such a response, you can just exit from the function without deleting the message from the queue. The message will become visible again after 30 seconds, and would trigger a function. 30 seconds is the default value of visibilitytimeout and is customizable on a per-msg basis.
Option 2: Distributed locks
I don't know from top of my head a distributed locking solution with counters. An alternative would be to implement a locking solution on your own using a SQL db and atomic transactions. A function when processing a message from Queue X will look into the db to see if a lock counter for X is less than N, and increase it by 1 if so, and then process the message. In this case, you will have to make sure that the locks get released even if your function crashes. That is, implement locks with lease expiration time.
来源：https://stackoverflow.com/questions/56955514/ensuring-no-more-than-n-instances-are-scaled-out-for-a-specific-queue