问题
I am trying to deploy spark job by using spark-submit
which has bunch of parameters like
spark-submit --class Eventhub --master yarn --deploy-mode cluster --executor-memory 1024m --executor-cores 4 --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"
I was looking a way to put all these flags in file to pass to spark-submit
to make my spark-submit
command simple liek this
spark-submit --class Eventhub --master yarn --deploy-mode cluster --config-file my-app.cfg --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"
anyone know if this is possible ?
回答1:
You can use --properties-file
which should include parameters with starting keyword spark
like
spark.driver.memory 5g
spark.executor.memory 10g
And command should look like:
spark-submit --class Eventhub --master yarn --deploy-mode cluster --properties-file <path-to-your-conf-file> --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"
来源:https://stackoverflow.com/questions/42835971/spark-submit-config-through-file