Spark 1.5.1 not working with hive jdbc 1.2.0

Question

I am trying to execute hive query using spark 1.5.1 in standalone mode and hive 1.2.0 jdbc version.

Here is my piece of code:

private static final St         


        

Answer 1

As far as I can tell, you're unfortunately out of luck in terms of using the jdbc connector until it's fixed upstream; the "Method not supported" in this case is not just a version mismatch, but is explicitly not implemented in the hive jdbc library branch-1.2 and even if you look at the hive jdbc master branch or branch-2.0 it's still not implemented:

public boolean isSigned(int column) throws SQLException {
  throw new SQLException("Method not supported");
}

Looking at the Spark callsite, isSigned is called during resolveTable in Spark 1.5 as well as at master.

That said, most likely the real reason this "issue" remains is that when interactive with Hive, you're expected to connect to the Hive metastore directly rather than needing to mess around with jdbc connectors; see the Hive Tables in Spark documentation for how to do this. Essentially, you want to think of Spark as an equal/replacement of Hive rather than being a consumer of Hive.

This way, pretty much all you do is add hive-site.xml to your Spark's conf/ directory and make sure the datanucleus jars under lib_managed/jars are available to all Spark executors, and then Spark talks directly to the Hive metastore for schema info and fetches data directly from your HDFS in a way amenable to nicely parallelized RDDs.