What does Exception: Randomness of hash of string should be disabled via PYTHONHASHSEED mean in pyspark?

Question

I am trying to create a dictionary from a list in pyspark. I have the following list of lists:

rawPositions

Gives

[[100979         


        

Answer 1

Check in Spark Configuration https://spark.apache.org/docs/latest/configuration.html#loading-default-configurations Runtime Environment part.

When running:

$SPARK_HOME/bin/spark-submit

Add:

--conf spark.executorEnv.PYTHONHASHSEED=321