teradata export from hive to teradata class not found

◇◆丶佛笑我妖孽 提交于 2021-02-08 06:50:51

问题


I am trying to export from hive table to teradata using TDCH connector, I am getting below errror:-

15/05/07 08:01:03 INFO tool.ConnectorExportTool: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
            at java.lang.Class.forName0(Native Method)
            at java.lang.Class.forName(Class.java:190)
            at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:81)
            at com.teradata.connector.common.tool.ConnectorExportTool.run(ConnectorExportTool.java:61)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
            at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:24)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
            at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
            at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
            ... 13 more

I understand from error that hive-metastore jar is missing. But it is already there in hive/lib folder

 hive-metastore.jar -> hive-metastore-0.9.0.jar

Is already present in the path: /usr/hdp/2.2.4.2-2/hive/lib


回答1:


You have to define environment variables LIB_JARS and HADOOP_CLASSPATH. You can use the environment variable LIB_JARS with the parameter -libjars

You can find an example under https://developer.teradata.com/sites/all/files/Teradata%20Connector%20for%20Hadoop%20Tutorial%20v1%200%20final.pdf.

According to the README, following jars are necessairy:

    Hive Job(version 0.11.0 as example):
         a) hive-metastore-0.11.0.jar
         b) hive-exec-0.11.0.jar
         c) hive-cli-0.11.0.jar
         d) libthrift-0.9.0.jar
         e) libfb303-0.9.0.jar
         f) jdo2-api-2.3-ec.jar
         g) slf4j-api-1.6.1.jar
         h) datanucleus-core-3.0.9.jar
         i) datanucleus-rdbms-3.0.8.jar
         j) commons-dbcp-1.4.jar
         k) commons-pool-1.5.4.jar
         l) antlr-runtime-3.4.jar
         m) datanucleus-api-jdo-3.0.7.jar

    HCatalog Job:
         a) above Hive required jar files
         b) hcatalog-core-0.11.0.jar



回答2:


The problem is that org/apache/hadoop/hive/metastore/api/MetaException class is not available to java runtime. can you ensure that the Jar is under valid runtime classpath.

few try outs can be try passing the classpath explicitly to jvm via -cp , alternatively you can try putting the jar in launch directory to ensure its present.




回答3:


Sometimes you may have all the jar files in your "lib" folder, but for some reason Oozie may not be reading it simply because you have a typo in your job.properties or coordinator.properties. Double check your properties file.

The "job.properties" file should read like the following -

oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/apps/myapp/workflow/

The "coordinator.properties" file should read like the following -

oozie.use.system.libpath=true
oozie.coord.application.path=${nameNode}/apps/myapp/workflow/

and your "lib" folder with jars should be at -

/apps/myapp/workflow/lib


来源:https://stackoverflow.com/questions/30100592/teradata-export-from-hive-to-teradata-class-not-found

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!