Can't get a SparkContext in new AWS EMR Cluster

Question

i just set up an AWS EMR Cluster (EMR Version 5.18 with Spark 2.3.2). I ssh into the master maschine and run spark-shell or pyspark and get the following error:

Answer 1

in order to fix this issue you can add configuration in json format on emr provisioning. We use code like this

{
    "Classification": "yarn-site",
    "Configurations": [
    ],
    "Properties": {
      "spark.yarn.app.container.log.dir": "/var/log/hadoop-yarn"
    }
  }