spark-submit config through file

Question

I am trying to deploy spark job by using spark-submit which has bunch of parameters like

spark-submit --class Eventhub --master yarn --deploy-mode cluster --executor-memory 1024m --executor-cores 4 --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"

I was looking a way to put all these flags in file to pass to spark-submit to make my spark-submit command simple liek this

spark-submit --class Eventhub --master yarn --deploy-mode cluster --config-file my-app.cfg --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"

anyone know if this is possible ?

Answer 1

You can use --properties-file which should include parameters with starting keyword spark like

spark.driver.memory 5g
spark.executor.memory 10g

And command should look like:

spark-submit --class Eventhub --master yarn --deploy-mode cluster --properties-file <path-to-your-conf-file> --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"