Apache Spark with Python: error

Question

New to Spark. Downloaded everything alright but when I run pyspark I get the following errors:

Type \"help\", \"copyright\", \"credits\" or \"license\" for more          


        

Answer 1

If you are doing it from the pyspark console, it may be because your installation did not work.

If not, it's because most example assume you are testing code in the pyspark console where a default variable 'sc' exist.

You can create a SparkContext by yourself at the beginning of your script using the following code:

from pyspark import SparkContext, SparkConf

conf = SparkConf()
sc = SparkContext(conf=conf)