Dynamically change maxWorkers with DeployerPartitionHandler

我们两清 提交于 2021-01-29 13:32:25

问题


Based on no. of partitions returned can the maxWorkers changed dynamically(at runtime) when using DeployerPartitionHandler

Regards, Balu

UPDATE Please find my use case. The batch execution starts for a normal business day with maxWorkers as "4" and partition returned as 40 but suddenly the load increases and that particular run of the batch returned 4 times the normal partitions (160). Here how do I increase the maxWorkers (say 16). Also if I start the batch always with a very high limit for maxWorkers (say 16), I believe the workers for a normal load can also hit 16 as each partition is processed by a worker, right ?


回答1:


They cannot. The idea around maxWorkers is really to set a high water mark that the app shouldn't go past. The dynamic aspect of the number of workers really is expected to come from the number of partitions. If you provide more detail in your question as to the specific use case, we can see if that still makes sense.

Update based on use case:

In your specific use case, you would need to have a listener (probably a JobExecutionListener) that would determine how many the max should be. What you have described is that you want a "regular max" and a "high load max". The listener would run before the step in question and set the max value in the ExecutionContext. Then, if the DeployerPartitionHandler is step scoped, you can inject the value from the ExecutionContext into the handler when that step executes.



来源:https://stackoverflow.com/questions/64560583/dynamically-change-maxworkers-with-deployerpartitionhandler

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!