Apache Spark with Python: error

Question

New to Spark. Downloaded everything alright but when I run pyspark I get the following errors:

Type \"help\", \"copyright\", \"credits\" or \"license\" for more          


        

Answer 1

It looks like you've found the answer to the second part of your question in the above answer, but for future users getting here via the 'org.apache.spark.sql.hive.HiveSessionState' error, this class is found in the spark-hive jar file, which does not come bundled with Spark if it isn't built with Hive.

You can get this jar at:

http://central.maven.org/maven2/org/apache/spark/spark-hive_${SCALA_VERSION}/${SPARK_VERSION}/spark-hive_${SCALA_VERSION}-${SPARK_VERSION}.jar

You'll have to put it into your SPARK_HOME/jars folder, and then Spark should be able to find all of the Hive classes required.