Enable case sensitivity for spark.sql globally

Question

The option spark.sql.caseSensitive controls whether column names etc should be case sensitive or not. It can be set e.g. by

spark_session.sql(\'         


        

Answer 1

Yet another way for PySpark. Using a SparkSession object named spark:

spark.conf.set('spark.sql.caseSensitive', True)

Answer 2

As it turns out setting

spark.sql.caseSensitive: True

in $SPARK_HOME/conf/spark-defaults.conf DOES work after all. It just has to be done in the configuration of the Spark driver as well, not the master or workers. Apparently I forgot that when I last tried.

Answer 3

Try sqlContext.sql("set spark.sql.caseSensitive=true") in your Python code, which worked for me.