Hadoop on Windows Building/ Installation Error

Question

I am trying to install Apache Hadoop 2.7.1 on Windows x64 (8.1 and Server 2012 R2), but I am failing at goal:

    [INFO] Apache Hadoop Common ..............         


        

Answer 1

I'd like to share my experience in building Hadoop 3.1.2 on Windows 8.1. I spent a lot of time fixing build errors, but all I needed was 'Building on Windows' section in Building.txt file in root folder of Hadoop sources.

I used "x64 Native Tools Command Prompt for VS 2017".

So, here are my steps

1. Read '\hadoop-3.1.2-src\Building.txt', section 'Building on Windows'.

This actually may be sufficient to figure out what you need to do, but with VS 2017 the process is a bit different.

2. Put sources into short path to avoid 'the command line is too long' error.

I used d:\hdp\ folder

3. As it's stated in Building.txt, set up environment using '\hadoop-3.1.2-src\dev-support\win-paths-eg.cmd'.

When you use Native Tools, you don't need a CALL command at the end of the .cmd file, so comment it out:

@REM CALL "%MSVS%\VC\vcvarsall.bat" %VCVARSPLAT%

My settings looks like this:

SET Platform=x64
SET VCVARSPLAT=amd64

@REM ******************
@REM Forcibly move the Maven local repo
@REM (use short path to avoid 'the command line is too long' error)

SET MAVEN_OPTS=-Dmaven.repo.local=D:\.m2

@REM *******************************************
@REM
@REM Locations of your bits and pieces
@REM
@REM NOTE: cmake is assumed to already be on the
@REM command path
@REM (it's true when you use Native Tools Command Prompt)

SET MAVEN_HOME=%MAVEN_HOME%
SET JAVA_HOME=%JAVA_HOME%
@REM SET MSVS=C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional
SET PROTO_BIN=D:\tools\protoc-2.5.0-win32
SET GIT_HOME=C:\Program Files (x86)\Git
set ZLIB_HOME=D:\tools\zlib-1.2.11

SET PATH=%JAVA_HOME%\bin;%MAVEN_HOME%\bin;%PROTO_BIN%;%GIT_HOME%\bin;%PATH%

@REM CALL "%MSVS%\VC\vcvarsall.bat" %VCVARSPLAT%

Off course you need Java, Maven, ProtoBuf, Git and ZLib to be installed in the paths listed above.

4. Run "x64 Native Tools Command Prompt for VS 2017"

From within "x64 Native Tools Command Prompt for VS 2017" navigate to Hadoop's source folder, run your 'win-paths-eg.cmd' and start building with Maven:

d:
cd D:\hdp
D:\hdp\dev-support\win-paths-eg.cmd
mvn package -Pdist,native-win -DskipTests -Dtar

5. Wait ~30-40 minutes.

That's all, folks!