AWS EMR 5.11.0 - Apache Hive on Spark

Question

I am trying to setup Apache Hive on Spark on AWS EMR 5.11.0. Apache Spark Version - 2.2.1 Apache Hive Version - 2.3.2 Yarn logs show below error:

18/01/28 21:55:28

Answer 1

I be able run hive on spark by run it like:

HIVE_AUX_JARS_PATH=$(find /usr/lib/spark/jars/ -name '*.jar' -and -not -name '*slf4j-log4j12*' -printf '%p:' | head -c-1) hive

Then, before other SQL queries issue:

SET hive.execution.engine = spark;

To make that persistent

Add line

export HIVE_AUX_JARS_PATH=$(find /usr/lib/spark/jars/ -name '*.jar' -and -not -name '*slf4j-log4j12*' -printf '%p:' | head -c-1)

into /home/hadoop/.bashrc

And in file /etc/hive/conf/hive-site.xml set:

  hive.execution.engine
  spark