hadoop native 冲突报错

僤鯓⒐⒋嵵緔 提交于 2020-05-08 17:18:33

问题描述

集群升级后,hadoop不能正常加载本地库

$ hadoop checknative -a
20/05/08 14:32:11 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
20/05/08 14:32:11 WARN zlib.ZlibFactory: Failed to load/initialize native-zlib library
20/05/08 14:32:11 ERROR snappy.SnappyCompressor: failed to load SnappyCompressor
java.lang.NoSuchFieldError: clazz
        at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native Method)
        at org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57)
        at org.apache.hadoop.io.compress.SnappyCodec.isNativeCodeLoaded(SnappyCodec.java:82)
        at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:89)
20/05/08 14:32:11 WARN lz4.Lz4Compressor: java.lang.NoSuchFieldError: clazz
Native library checking:
hadoop:  true /home/hadoop/core/jdk/jre/lib/amd64/libhadoop.so
zlib:    false 
snappy:  false 
lz4:     true revision:99
bzip2:   false 
openssl: true /lib64/libcrypto.so
20/05/08 14:32:11 INFO util.ExitUtil: Exiting with status 1

查看旧版本的加载情况

$ cd hadoop-2.7.7/
$ ./bin/hadoop checknative -a
20/05/08 14:34:19 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
20/05/08 14:34:19 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/core/jdk/jre/lib/amd64/libhadoop.so
zlib:    true /lib64/libz.so.1
snappy:  true /home/hadoop/core/hadoop-2.7.7/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so

发现旧版本可以加载,但是本地库的路径似乎有所不一样,hadoop的加载到了jdk的目录下,snappy的加载到hadoop目录下

解决办法

根据上面的情况发现,尝试删掉jdk下的库文件

$ mv /home/hadoop/core/jdk/jre/lib/amd64/libhadoop.so /home/hadoop/core/jdk/jre/lib/amd64/libhadoop.so.bak

再次查看,发现可以加载了

$ hadoop checknative -a
20/05/08 14:43:08 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
20/05/08 14:43:08 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /opt/beh/core/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /opt/beh/core/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so

问题分析

加载库的路径不对导致的冲突,主要在于集群做的调整太多了,对接了多种其它的文件系统,本来还可以正常运行,但是做了版本升级,那先前做的更改可能就会对高版本不太适用了。

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!