Oozie keep adding a old version of httpcore jar to classpath

问题

I'm keep getting an exception because Oozie add a wrong version of httpcore jar to classpath. I tryed different options such as

oozie.launcher.mapreduce.task.classpath.user.precedence
oozie.launcher.mapreduce.user.classpath.first

oozie.launcher.mapreduce.task.classpath.user.precedence does not respond at all and when I use oozie.launcher.mapreduce.user.classpath.first, application cannot load even one class.

In class path I can see two versions of http-core.

httpcore-4.4.1.jar
httpcore-4.2.4.jar

When application runs in stand alone mode, I'm not getting that exception.

Exception:

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchFieldError: INSTANCE
org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.createConnectionSocketFactoryRegistry(ExchangeServiceBase.java:244)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.initializeHttpClient(ExchangeServiceBase.java:198)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:174)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:179)
    at microsoft.exchange.webservices.data.core.ExchangeService.<init>(ExchangeService.java:3729)
    at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:16)
    at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:10)
    at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.<init>(EwsOperations.java:47)
    at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.getInstance(EwsOperations.java:53)
    at com.sonasoft.sonacloud.email.dispatcher.main.MainClass.main(MainClass.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
    ... 15 more

Oozie client build version: 4.2.0.2.3.2.0-2950

Any help is appreciated.

回答1:

We have had this nasty issue with HortonWorks distro 2.3.2 (shame on them):

the Oozie "launcher" job always gets httpcore and httpclient in the CLASSPATH as part of the Hadoop client
the Oozie "launcher" job always gets httpcore and httpclient as bundled in the "Oozie" ShareLib
the Hive/Hive2 Sharelibs contain httpcore and httpclient in a more recent version
from Hadoop point of view, user.classpath.first applies to both ShareLibs so it's a 50/50 chance of getting the right order for each JAR (so a 25/75 chance overall)

Bottom line: we had to

remove httpcore and httpclient from the "Oozie" ShareLib HDFS dir (duh!)
raise oozie.launcher.mapreduce.job.user.classpath.first flag for all actions relying on Hive JARS (i.e. Hive action, Hive2 action, Shell action calling the JDBC driver somehow, etc.)

Post-scriptum -- the Oozie server keeps in memory a list of the JARs in each ShareLib, so that removing a JAR while the server is running will trigger errors in new jobs. If you don't want to stop the Oozie server, then the "proper way" to update a live ShareLib is to (a) create a new version in a new, time-stamped directory [check the documentation...] and (b) tell the server to resync on the newer libs with oozie admin -sharelibupdate

回答2:

You want to build with your local version of the httpcore JAR, but you don't want it present in your classpath, because Hadoop will provide its own version. Then you should be using the provided scope for the httpcore JAR:

<project>
    ...
    <dependencies>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <scope>provided</scope>           <!-- this line is important -->
            <version>4.4.1</version>
        </dependency>
    </dependencies>
</project>

From the Maven documentation for provided:

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime.

来源：https://stackoverflow.com/questions/34869346/oozie-keep-adding-a-old-version-of-httpcore-jar-to-classpath

标签

java

Hadoop

Oozie