How to set Spark configuration properties using Apache Livy?

问题

I don't know how to pass SparkSession parameters programmatically when submitting Spark job to Apache Livy:

This is the Test Spark job:

class Test extends Job[Int]{

  override def call(jc: JobContext): Int = {

    val spark = jc.sparkSession()

    // ...

  }
}

This is how this Spark job is submitted to Livy:

val client = new LivyClientBuilder()
  .setURI(new URI(livyUrl))
  .build()

try {
  client.uploadJar(new File(testJarPath)).get()

  client.submit(new Test())

} finally {
  client.stop(true)
}

How can I pass the following configuration parameters to SparkSession?

  .config("es.nodes","1localhost")
  .config("es.port",9200)
  .config("es.nodes.wan.only","true")
  .config("es.index.auto.create","true")

回答1:

You can do that easily through the LivyClientBuilder like this:

val client = new LivyClientBuilder()
  .setURI(new URI(livyUrl))
  .setConf("es.nodes","1localhost")
  .setConf("key", "value")
  .build()

回答2:

Configuration parameters can be set to LivyClientBuilder using

public LivyClientBuilder setConf(String key, String value)

so that your code starts with:

val client = new LivyClientBuilder()
  .setURI(new URI(livyUrl))
  .setConf("es.nodes","1localhost")
  .setConf("es.port",9200)
  .setConf("es.nodes.wan.only","true")
  .setConf("es.index.auto.create","true")
  .build()

来源：https://stackoverflow.com/questions/49224564/how-to-set-spark-configuration-properties-using-apache-livy