Hadoop on Windows Building/ Installation Error

Question

I am trying to install Apache Hadoop 2.7.1 on Windows x64 (8.1 and Server 2012 R2), but I am failing at goal:

    [INFO] Apache Hadoop Common ..............         


        

Answer 1

I've managed to build it with Visual Studio 2015 community edition.

Here's how I've built it:

My environment

Here's my shopping list:

• Windows 10
• JDK 1.8.0_51
• Maven 3.3.3
• Findbugs 1.3.9 (I haven't used this)
• ProtocolBuffer 2.5.0 (I didn't pick the latest and greatest here - it has to be 2.5.0)
• CMake 3.3.0
• Visual Studio 2015 Community Edition
• GnuWin32 0.6.3 - a bit painful to install but so is cygwin
• zlib 1.2.8
• internet connection

Windows System Environment variables

• JAVA_HOME = "C:\Program Files\Java\jdk1.8.0_51"
• MAVEN_HOME=c:\apache-maven-3.3.3

(make sure you point the above to your JDK version and maven installation)

I appended the following to my windows system environment Path variable:

;%MAVEN_HOME%\bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;c:\zlib

The weird "C:\Windows\Microsoft.NET\Framework64\v4.0.30319" path is the location of MSBuild.exe, which is required during the build process.

Protoc Buffers 2.5.0

Oh no, another unix/linux only build? I've downloaded the google package named protoc-2.5.0-win32.zip. Then extracted the binary file (protoc.exe) to c:\windows\system32 - just a lazy way to put it on the path.

I'm not 100% sure of the effect of having a win32 component for this win64 build. But: "Hadoop 0.23+ requires the protocol buffers JAR (protobufs.jar) to be on the classpath of both clients and servers; the native binaries are required to compile this and later versions of Hadoop." - http://wiki.apache.org/hadoop/ProtocolBuffers.

So I understand the win32 executable is used only during the build process (the jar equivalent should be packaged in the build).

If it is used in any way to compile native code, we may have left with some pointers out of order. I'll come back to this when I can.

Tweaking the Hadoop sources

Well, this was necessary to allow to build to execute. It shouldn't affect the quality of the build itself, but let's keep in mind the result is an unofficial, unsupported, use at your own risk hadoop, intended for a development environment.

Migrating VS projects

The following files need to be open with Visual Studio 2015:

\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj \hadoop-common-project\hadoop-common\src\main\native\native.vcxproj

Visual Studio will complain of them being of an old version. All you have to do is to save all and close.

Enabling cmake VS 2015 project generation for hdfs

On the line 441 of \hadoop-hdfs-project\hadoop-hdfs\pom.xml, edit the else value as the following:

<condition property="generator" value="Visual Studio 10" else="Visual Studio 14 2015 Win64">

(the "value" value applies to win32 - you may want to edit it if building for win32).

Building it

You should try and find on windows the "Development Command Prompt for VS2015". I'm still wondering what is so special about this, but the fact is that it will only work with that.

More Environment variables Those should be done on the command prompt:

set Platform=x64

set ZLIB_HOME=C:\zlib\include (unlike the official instructions, this should be pointing to the include folder).

Finally building it

Go to the hadoop source folder and issue:

mvn package -Pdist,native-win -DskipTests -Dtar

What next?

Follow the official docs to get your hadoop instance configured and up and running.

I'll try to keep a link with for the binaries on my blog:

http://kplitzkahran.blogspot.co.uk/2015/08/hadoop-271-for-windows-10-binary-build.html

Answer 2

I successfully built Hadoop 3.0.2 with Visual Studio 2017 on Windows 10.

Here is my settings look like:

1. Windows 10
2. Visual Studio 2017 Community Version
3. Maven 3.5.0
4. JDK 1.8.0_144
5. Cmake 3.10.0-rc1
6. ProtocolBuffer 2.5.0
7. Cygwin

First I open these two solution files in VS 2017 and save.

hadoop-3.0.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln

and

hadoop-3.0.2-src\hadoop-common-project\hadoop-common\src\main\native\native.sln

This will migrate those two projects from VS 2010 to VS 2017.

Second I need to change line 141 of the pom file located at

D:\hadoop-3.0.2-src\hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml

to

<condition property="generator" value="Visual Studio 10" else="Visual Studio 15 2017 Win64">

This will ensure VS 2017 will be used during the building process.

I have also encountered a problem where MSBuild complains the command is long then stops working, I think this is related to the Maven default repository location. To fix this, I added following line

<localRepository>D:/maven_repo</localRepository>

to file

apache-maven-3.5.0\conf\settings.xml

to move the Maven repository to a new location (in this case D:/maven_repo).

Finally I opened the x64 Native Tools Command Prompt for VS 2017, cd to the hadoop source folder and issued following command:

mvn clean package -Pdist,native-win -DskipTests -Dtar -Dmaven.repo.local=D:\maven_repo

I spend more than 10 hours to fix all problems, hopefully my solution will help others as well.

Answer 3

I'd like to share my experience in building Hadoop 3.1.2 on Windows 8.1. I spent a lot of time fixing build errors, but all I needed was 'Building on Windows' section in Building.txt file in root folder of Hadoop sources.

I used "x64 Native Tools Command Prompt for VS 2017".

So, here are my steps

1. Read '\hadoop-3.1.2-src\Building.txt', section 'Building on Windows'.

This actually may be sufficient to figure out what you need to do, but with VS 2017 the process is a bit different.

2. Put sources into short path to avoid 'the command line is too long' error.

I used d:\hdp\ folder

3. As it's stated in Building.txt, set up environment using '\hadoop-3.1.2-src\dev-support\win-paths-eg.cmd'.

When you use Native Tools, you don't need a CALL command at the end of the .cmd file, so comment it out:

@REM CALL "%MSVS%\VC\vcvarsall.bat" %VCVARSPLAT%

My settings looks like this:

SET Platform=x64
SET VCVARSPLAT=amd64

@REM ******************
@REM Forcibly move the Maven local repo
@REM (use short path to avoid 'the command line is too long' error)

SET MAVEN_OPTS=-Dmaven.repo.local=D:\.m2

@REM *******************************************
@REM
@REM Locations of your bits and pieces
@REM
@REM NOTE: cmake is assumed to already be on the
@REM command path
@REM (it's true when you use Native Tools Command Prompt)

SET MAVEN_HOME=%MAVEN_HOME%
SET JAVA_HOME=%JAVA_HOME%
@REM SET MSVS=C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional
SET PROTO_BIN=D:\tools\protoc-2.5.0-win32
SET GIT_HOME=C:\Program Files (x86)\Git
set ZLIB_HOME=D:\tools\zlib-1.2.11

SET PATH=%JAVA_HOME%\bin;%MAVEN_HOME%\bin;%PROTO_BIN%;%GIT_HOME%\bin;%PATH%

@REM CALL "%MSVS%\VC\vcvarsall.bat" %VCVARSPLAT%

Off course you need Java, Maven, ProtoBuf, Git and ZLib to be installed in the paths listed above.

4. Run "x64 Native Tools Command Prompt for VS 2017"

From within "x64 Native Tools Command Prompt for VS 2017" navigate to Hadoop's source folder, run your 'win-paths-eg.cmd' and start building with Maven:

d:
cd D:\hdp
D:\hdp\dev-support\win-paths-eg.cmd
mvn package -Pdist,native-win -DskipTests -Dtar

5. Wait ~30-40 minutes.

That's all, folks!

Answer 4

Took me the whole day to manage to build it... Microsoft has obviously lots of issues with backward compatibility. It would help if Hadoop would move to a more recent toolchain.

Here are the key steps I had to perform (beyond those mentioned in the doc). This is using the Windows SDK compiler rather than VS 2010 (it is supposed to be possible with VS 2010 but I haven't tried it):

1. Uninstall Visual Studio 2013 (I would recommend uninstalling any Visual Studio >= 2010) -- this is to get rid of error MSBUILD : Configuration error MSB4146: Cannot evaluate the property expression "$([MSBuild]::ValueOrDefault('$(VCTargetsPath)','$(MSBuildExtensionsPath32)\Microsoft.Cpp\v4.0\V120\'))"
2. Uninstall Visual Studio C++ 2010 Redistributable (x64 + x86) -- to avoid Error when installing windows SDK 7.1
3. Uninstall fully .NET Framework 4.5 -- to avoid Failure during conversion to COFF: file invalid or corrupt
4. Reboot
5. Install .NET Framework 4.0
6. Install Windows SDK 7.1

I added to my PATH (in the Windows SDK shell) the bin folder of my Git installation (for GNU tools), the bin folder of CMake, and the folder containing protoc.exe. In addition (but most people should not need it), I had to setup Maven to use my HTTP proxy, first editing settings.xml as indicated in Maven's documentation, but also by defining the environment variable MAVEN_OPTS=-Dmy.proxy.host -Dhttp.proxyPort=my.proxy.port, as otherwise there was a timeout at some point during the build.

Note that I noticed a good amount of warnings during the build: I suspect some of them to be caused by using the Java SDK 1.8 instead of 1.7. However after finally seeing a BUILD SUCCESS I didn't feel like trying again with 1.7.