问题
I am trying to run a Insert in to partition table with group by involved
query
'set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.execution.engine=tez;
INSERT OVERWRITE TABLE table1 PARTITION (date) select col1,CONCAT(COALESCE(substr(Cdate,1,4),'-'),'',COALESCE(substr(Cdate,6,2),'-'),'',COALESCE(substr(Cdate,9,2),'-')),col3,col4,'mobile-data',data,date
from
(select col1,substr(CDate,1,10) as Cdate,u.col3 as col3,u.col4 as col4,date,sum(u.col5+u.col6) as data from json_table lateral view explode(json_table.mobiledata) oc as u
where date >= from_unixtime(unix_timestamp()-4*60*60*24, 'yyyyMMdd')
group by col1,substr(CDate,1,10),u.col3,u.col4,date)t;'
error:
'Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x48x56x49x57x0x1x117x115x101x114x49x0x1x129x51x122x35x1x65x78x84x32x72x65x76x32x83x101x114x118x105x99x101x0x1x99x111x109x46x100x115x105x46x97x110x116x46x115x101x114x118x101x114x0x1x109x111x98x105x108x101x45x100x97x116x97x0x255 with properties{columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=++++++, columns.types=string,string,string,string,string,string}'
I have never seen this error. Does anyone know how to take care of this?
This query is working fine if i just use the select part of the query excluding the insert into partitioned table.
回答1:
I was able to get this resolved using help from this link
http://documentation.altiscale.com/hive-runtime-error-unable-to-deserialize-input-key
All i needed was a setting change.
'Set hive.optimize.sort.dynamic.partition = false;'
Once i add this setting, query(insert into partitioned table) is working fine.
来源：https://stackoverflow.com/questions/32236798/hive-runtime-error-unable-to-deserialize-reduce-input-key