How to set spark.driver.memory for Spark/Zeppelin on EMR

问题

When using EMR (with Spark, Zeppelin), changing spark.driver.memory in Zeppelin Spark interpreter settings won't work.

I wonder what is the best and quickest way to set Spark driver memory when using EMR web interface (not aws CLI) to create clusters?

Is Bootstrap action could be a solution? If yes, can you please provide an example of how the bootstrap action file should look like?

回答1:

You can always try to add the following configuration on job flow/cluster creation :

[
    {
        "Classification": "spark-defaults",
        "Properties": {
            "spark.driver.memory": "12G"
        }
    }
]

You can do this most of the configurations whether for spark-default, hadoop core-site, etc.

I hope this helps !

来源：https://stackoverflow.com/questions/47531410/how-to-set-spark-driver-memory-for-spark-zeppelin-on-emr