Add conf file to classpath in Google Dataproc

Question

We\'re building a Spark application in Scala with a HOCON configuration, the config is called application.conf.

If I add the application.conf

Answer 1

If --jars doesn't work as suggested in this answer, you can try init action. First upload your config to GCS, then write an init action to download it to the VMs, putting it to a folder in the classpath or update spark-env.sh to include the path to the config.