Hadoop “Unable to load native-hadoop library for your platform” warning

Question

I\'m currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error:

Answer 1

For installing Hadoop it is soooooo much easier installing the free version from Cloudera. It comes with a nice GUI that makes it simple to add nodes, there is no compiling or stuffing around with dependencies, it comes with stuff like hive, pig etc.

http://www.cloudera.com/content/support/en/downloads.html

Steps are: 1) Download 2) Run it 3) Go to web GUI (1.2.3.4:7180) 4) Add extra nodes in the web gui (do NOT install the cloudera software on other nodes, it does it all for you) 5) Within the web GUI go to Home, click Hue and Hue Web UI. This gives you access to Hive, Pig, Sqoop etc.

Answer 2

This also would work:

export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native

Answer 3

In addition to @zhutoulala accepted answer, here is an update to make it work with latest stable version to date (2.8) on ARMHF platforms (Raspberry Pi 3 model B). First I can confirm that you must recompile native libraries to 64 bit ARM, other answers here based on setting some environment variables won't work. As indicated in Hadoop documentation, the pre-built native libraries are 32 bit.

High level steps given in the fist link (http://www.ercoppa.org/posts/how-to-compile-apache-hadoop-on-ubuntu-linux.html) are correct. On this url http://www.instructables.com/id/Native-Hadoop-260-Build-on-Pi/ you get more details specific to Raspberry Pi, but not for Hadoop version 2.8.

Here are my indications pour Hadoop 2.8 :

• there is still no protobuf package on latest Raspbian so you must compile it yourself and version must be exactly protobuf 2.5 (https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz)

• CMake file patching method must be changed. Moreovere, files to patch are not the same. Unfortunately, there is no accepted patch on JIRA specific to 2.8. On this URL (https://issues.apache.org/jira/browse/HADOOP-9320) you must copy and paste Andreas Muttscheller proposed patch on your namenode :

  :hadoop-2.8.0-src/hadoop-common-project/hadoop-common $ touch HADOOP-9320-v2.8.patch
  :hadoop-2.8.0-src/hadoop-common-project/hadoop-common $ vim HADOOP-9320-v2.8.patch
  #copy and paste proposed patch given here : https://issues.apache.org/jira/browse/HADOOP-9320?focusedCommentId=16018862&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16018862
  :hadoop-2.8.0-src/hadoop-common-project/hadoop-common $ patch < HADOOP-9320-v2.8.patch
  patching file HadoopCommon.cmake
  patching file HadoopJNI.cmake
  :hadoop-2.8.0-src/hadoop-common-project/hadoop-common $ cd ../..
  :hadoop-2.8.0-src $ sudo mvn package -Pdist,native -DskipTests -Dtar
  

Once build is successful :

    :hadoop-2.8.0-src/hadoop-dist/target/hadoop-2.8.0/lib/native $ tar -cvf nativelibs.tar *

And replace the content of the lib/native directory of your Hadoop install with the content of this archive. Warning message when running Hadoop should disappear.