How do I fix this Kryo exception when using a UDF on hive?

问题

I have a hive query that worked in hortonworks 2.6 sandbox, but it doesn't work on sandbox ver. 3.0 because of this exception:

Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 95                                                                                                          
Serialization trace:                                                                                                                                                                                               
parentOperators (org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkLongOperator)                                                                                                                    
childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator)                                                                                                                                        
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)                                                                                                                                                  
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)                                                                                                                                                               
        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)                                                                                            
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)                                                                                                                                 
        at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:185)

How do I fix it?

I have seen some answers suggesting doing set hive.exec.parallel=false; but it doesn't work, I still get this error.

I checked the versions of libraries that I use and made sure that hadoop version and hive --version match the versions of libraries that I use in my jar.

I also tried this: https://community.hortonworks.com/content/supportkb/150199/orgapachehivecomesotericsoftwarekryokryoexception-1.html it did not work either.

回答1:

I was finally able to run my queries after I reduced the size of my udf.jar. It used to be 150 mb and I reduced it to 50 kb. It seems like a kryo bug. I got this info from here: https://github.com/EsotericSoftware/kryo/issues/307

I reduced the size of my udf.jar by marking all dependencies as provided. So I went from this:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>3.1.1</version>
</dependency>

to this:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>3.1.1</version>
    <scope>provided</scope> <!--Notice this line-->
</dependency>

This is definitely a kryo bug, because I was able to run this query with that large udf.jar file in hortonworks 2.6.

I hope someone finds this information valuable.

回答2:

I actually referred to this thread of hive https://issues.apache.org/jira/browse/HIVE-7711 the comment by Dr.Christian Betz on this issue suggests implementing DoNothingSerializer to solve this exception. I tried implementing that solution and the problem was solved. However, I could not understand the details of how adding Serializer is solving this issue. Find the solution at this link. Try implementing following Serializer for the class which extends GenericUDF.

import org.apache.hive.com.esotericsoftware.kryo.Kryo;

import org.apache.hive.com.esotericsoftware.kryo.Serializer;

import org.apache.hive.com.esotericsoftware.kryo.io.Input;

import org.apache.hive.com.esotericsoftware.kryo.io.Output;



public class DoNothingSerializer extends Serializer<App> {



@Override

public App read(Kryo arg0, Input arg1, Class<App> arg2) {

// TODO Auto-generated method stub

return new App();

}



@Override

public void write(Kryo arg0, Output arg1, App arg2) {

// TODO Auto-generated method stub



}



}

来源：https://stackoverflow.com/questions/54572121/how-do-i-fix-this-kryo-exception-when-using-a-udf-on-hive

标签

Hive

hortonworks-sandbox

apache-tez