问题
Usecase => Create two YARN queues: Q1 and Q2 with the configuration below.
[
{
"Classification": "capacity-scheduler",
"Properties": {
"yarn.scheduler.capacity.root.queues" : "Q1,Q2",
"yarn.scheduler.capacity.root.Q1.capacity" : "60",
"yarn.scheduler.capacity.root.Q2.capacity" : "40",
"yarn.scheduler.capacity.root.Q1.accessible-node-labels" : "*",
"yarn.scheduler.capacity.root.Q2.accessible-node-labels" : "*",
"yarn.scheduler.capacity.root.Q1.accessible-node-labels.CORE.capacity" : "60",
"yarn.scheduler.capacity.root.Q2.accessible-node-labels.CORE.capacity" : "40",
"yarn.scheduler.capacity.root.Q1.accessible-node-labels.CORE.maximum-capacity" : "60"
}
},
{
"Classification": "yarn-site",
"Properties": {
"yarn.acl.enable": "true",
"yarn.resourcemanager.scheduler.class": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler"
}
}
]
Expected behavior: Q1 should not use more than 60% percent of CORE nodes so that 40% is always available for Q2. See YARN doc to understand Queue configs. Another reference to understand max-capacity configuration in this book.
Actual behavior: Q1 uses more than 60%, i.e. "Absolute Used Capacity" for the queue "Q1" is greater than the "Absolute Configured Max Capacity".
This is not as per the YARN documentation. I would like to understand the cause behind this behavior. And alternatives solutions to this.
Update 1: This problems seems to be only with the CORE nodes. If I specify below, it works as expected for task nodes. yarn.scheduler.capacity.root.Q1.maximum-capacity: 60
EMR creates CORE nodes under the YARN node label as CORE. By default, EMR creates this node label. More on the YARN node labels and EMR 5.19.0 using the YARN node label feature. IMHO, while using YARN node label feature for CORE nodes, EMR is either over-riding or have broken this configuration for CORE nodes.
来源：https://stackoverflow.com/questions/60667585/yarn-queue-capacity-not-working-as-expected-for-core-nodes-on-emr-emr-5-26-0