Why number of executors shown in Spark Web UI and rest UI is constant always , when dynamic allocation is enabled

倖福魔咒の 提交于 2019-12-11 05:14:36

问题


I am running a Spark Streaming application with batch duration of 1 minute, and I have dynamic allocation enabled with executorIdleRunTime equal to 5 seconds. I can see in event timeline that they about 4000 many executors allocated and removed but in executors I always see 102 number of executors.What can be the reason of that.Rest API gives the same data but executor is there is different.

I have attached the respective snapshots for better understanding of the question.

来源:https://stackoverflow.com/questions/45012864/why-number-of-executors-shown-in-spark-web-ui-and-rest-ui-is-constant-always-w

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!