问题
When I start H2o on a cdh cluster I get the following error. I downloaded everything formt he wbesite and followed the tutorial. The command I ran was
hadoop jar h2odriver.jar -nodes 2 -mapperXmx 1g -output hdfsOutputDirName
It shows that containers are not being used. It's not clear what settings these would be on hadoop. I have given all settings memory. It's the 0.0 for memory that doesnt make sense, and why are the containers not using memory. Is the cluster even running now?
----- YARN cluster metrics -----
Number of YARN worker nodes: 3
----- Nodes -----
Node: http://data-node-3:8042 Rack: /default, RUNNING, 1 containers used, 1.0 / 6.0 GB used, 1 / 4 vcores used
Node: http://data-node-1:8042 Rack: /default, RUNNING, 0 containers used, 0.0 / 6.0 GB used, 0 / 4 vcores used
Node: http://data-node-2:8042 Rack: /default, RUNNING, 0 containers used, 0.0 / 6.0 GB used, 0 / 4 vcores used
----- Queues -----
Queue name: root.default
Queue state: RUNNING
Current capacity: 0.00
Capacity: 0.00
Maximum capacity: -1.00
Application count: 0
Queue 'root.default' approximate utilization: 0.0 / 0.0 GB used, 0 / 0 vcores used
----------------------------------------------------------------------
WARNING: Job memory request (2.2 GB) exceeds queue available memory capacity (0.0 GB)
WARNING: Job virtual cores request (2) exceeds queue available virtual cores capacity (0)
----------------------------------------------------------------------
For YARN users, logs command is 'yarn logs -applicationId application_1462681033282_0008'
回答1:
You should setup your default queue to have available resources to run 2nodes cluster.
See warnings:
WARNING: Job memory request (2.2 GB) exceeds queue available memory capacity (0.0 GB)
- you ask 1GB per node (+overhead) but there is no available resources in the YARN queue
WARNING: Job virtual cores request (2) exceeds queue available virtual cores capacity (0)
- you ask for 2 virtual cores but no cores are available in your default queue
Please check YARN documentation - for example setup of capacity scheduler and max available resources: https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
回答2:
I made the following changes in Cloudera Manager yarn configuration
Setting Value
yarn.scheduler.maximum-allocation-vcores 8
yarn.nodemanager.resource.cpu-vcores 4
yarn.nodemanager.resource.cpu-vcores 4
yarn.scheduler.maximum-allocation-mb 16 GB
来源:https://stackoverflow.com/questions/37096589/h2o-starting-on-yarn-not-working