I have a spark as interpreter in Zeppelin. I\'m using a Spark2.0, I built a Session: Create
In general you should not initialize SparkSession
nor SparkContext
in Zeppelin. Zeppelin notebooks are configured to create session for you, and their correct behavior depends on using provided objects.
Initializing your SparkSession
will break core Zeppelin functionalities, and multiple SparkContexts
will break things completely in the worst case scenario.
Is set spark.driver.allowMultipleContexts to False is best to do a tests ?
You should never use spark.driver.allowMultipleContexts
- it not supported, and doesn't guarantee correct results.