1. 技术文章
  2. Building Giraph with Hadoop

Building Giraph with Hadoop

由 大兔子大兔子 提交于 2019-12-01 05:46:19

问题


I am trying to set up Giraph with Hadoop 2.7.1 Try as I might, it doesn't seem to work.

I have tried following the below links,

  • http://giraph.apache.org/build.html
  • https://blog.cloudera.com/blog/2014/02/how-to-write-and-run-giraph-jobs-on-hadoop/
  • http://lab.hypotheses.org/1207

What happens when I try to build using mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.374 s]
[INFO] Apache Giraph Core ................................. FAILURE [  8.469 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.136 s
[INFO] Finished at: 2015-11-16T10:19:07+05:30
[INFO] Final Memory: 45M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-cor

What happens when I try to build using, mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ giraph-parent ---
[INFO] Installing /usr/local/giraph/pom.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0.pom
[INFO] Installing /usr/local/giraph/target/giraph-parent-1.1.0-site.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0-site.xml
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.360 s]
[INFO] Apache Giraph Core ................................. FAILURE [  7.473 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.122 s
[INFO] Finished at: 2015-11-16T10:26:04+05:30
[INFO] Final Memory: 46M/374M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-core







---
Version information follows,

Hadoop:  

    hadoopuser:~$ hadoop version
    Hadoop 2.7.1
    Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
    Compiled by jenkins on 2015-06-29T06:04Z
    Compiled with protoc 2.5.0
    From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

Giraph is version 1.1.0 downloaded from here : http://archive.apache.org/dist/giraph/giraph-1.1.0/giraph-dist-1.1.0-src.tar.bz2

Java

    hadoopuser:~$ java -version
    java version "1.8.0_05"
    Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

Maven 

    hadoopuser:~$ mvn -version
    Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T17:27:37+05:30)
    Maven home: /opt/apache-maven-3.3.3
    Java version: 1.8.0_05, vendor: Oracle Corporation
    Java home: /usr/lib/jvm/jdk1.8.0/jre
    Default locale: en_IN, platform encoding: UTF-8
    OS name: "linux", version: "3.13.0-68-generic", arch: "amd64", family: "unix"

Edit:

As was suggested by Steve Kaeser in his answer I changed my pom.xlm and changed part is shown below.

  <properties>
<hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
    <munge.symbols>PURE_YARN</munge.symbols>
    <!-- TODO: add these checks eventually -->
    <project.enforcer.skip>true</project.enforcer.skip>
    <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
    <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
  </properties>

After doing this when I run mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install I get a lot of output and it fails. Parts of the output are shown below,

[INFO] 
[INFO] --- apache-rat-plugin:0.8:check (default) @ giraph-core ---
[INFO] Exclude: CODE_CONVENTIONS
[INFO] Exclude: **/target/**
[INFO] Exclude: /_bsp/**
[INFO] Exclude: .checkstyle
[INFO] Exclude: .reviewboardrc
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: .arcconfig
[INFO] Exclude: *.patch
[INFO] Exclude: .idea/**
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.ipr
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: giraph-gora/conf/*
[INFO] Exclude: giraph-rexster/giraph-kibble/src/main/resources/META-INF/services/com.tinkerpop.rexster.extension.RexsterExtension
[INFO] Exclude: src/site/resources/images/*.svg
[INFO] 
[INFO] >>> findbugs-maven-plugin:2.5.1:check (default) > :findbugs @ giraph-core >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
[INFO] Fork Value is true
     [java] The following errors occurred during analysis:
     [java] Nov 17, 2015 3:16:36 PM edu.umd.cs.findbugs.TextUIBugReporter reportAnalysisError
     [java] SEVERE: Error scanning java/util/Comparator for referenced classes
     [java] java.lang.ArrayIndexOutOfBoundsException: 51966
     [java]     at org.objectweb.asm.ClassReader.readUTF8(Unknown Source)
     [java]     at org.objectweb.asm.ClassReader.readClass(Unknown Source)

.
.
.
.
.
.
.
.
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.140 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 37.541 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.972 s
[INFO] Finished at: 2015-11-17T15:16:37+05:30
[INFO] Final Memory: 68M/537M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs (findbugs) on project giraph-core: Execution findbugs of goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs failed: Java returned: 1 -> [Help 1]
[ERROR]

回答1:


I ran into this issue today. Finally I needed two patches to fix it:

  1. Open pom.xml in base directory and replace
    <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols> with:
    <munge.symbols>PURE_YARN</munge.symbols>

  2. Apply patch found here: GIRAPH-1110.02.patch

  3. Build using the clean flag of maven:
    mvn -Phadoop_yarn -Dhadoop.version=2.7.3 clean package -DskipTests




回答2:


I just ran into this issue earlier today and found a reply in the mailing list that suggests removing STATIC_SASL_SYMBOL from munge.symbols in the hadoop_yarn profile.

I was able to build that profile after making that change.

Here is the link to the mailing list archive: http://mail-archives.apache.org/mod_mbox/giraph-user/201501.mbox/%3C54B17196.4040107@hiro-tan.org%3E



来源：https://stackoverflow.com/questions/33728752/building-giraph-with-hadoop

标签
java
maven
Hadoop
build
giraph
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!