The memory consumption of hadoop's namenode?

Question

Can anyone give a detailed analysis of memory consumption of namenode? Or is there some reference material ? Can not find material in the network.Thank you!

Answer 1

I guess we should make the distinction between how namenode memory is consumed by each namenode object and general recommendations for sizing the namenode heap.

For the first case (consumption) ,AFAIK , each namenode object holds an average 150 bytes of memory. Namenode objects are files, blocks (not counting the replicated copies) and directories. So for a file taking 3 blocks this is 4(1 file and 3 blocks)x150 bytes = 600 bytes.

For the second case of recommended heap size for a namenode, it is generally recommended that you reserve 1GB per 1 million blocks. If you calculate this (150 bytes per block) you get 150MB of memory consumption. You can see this is much less than the 1GB per 1 million blocks, but you should also take into account the number of files sizes, directories.

I guess it is a safe side recommendation. Check the following two links for a more general discussion and examples:

Sizing NameNode Heap Memory - Cloudera

Configuring NameNode Heap Size - Hortonworks

Namenode Memory Structure Internals