Spark spark-submit --jars arguments wants comma list, how to declare a directory of jars?

Question

In Submitting Applications in the Spark docs, as of 1.6.0 and earlier, it\'s not clear how to specify the --jars argument, as it\'s apparently not a colon-separated classpat

Answer 1

One way (the only way?) to use the --jars argument is to supply a comma-separated list of explicitly named jars. The only way I figured out to use the commas was a StackOverflow answer that led me to look beyond the docs to the command line:

spark-submit --help 

The output from that command contains:

 --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths. 

Today when I was testing --jars, we had to explicitly provide a path to each jar:

/usr/local/spark/bin/spark-submit --class jpsgcs.thold.PipeLinkageData ---jars=local:/usr/local/spark/jars/groovy-all-2.3.3.jar,local:/usr/local/spark/jars/guava-14.0.1.jar,local:/usr/local/spark/jars/jopt-simple-4.6.jar,local:/usr/local/spark/jars/jpsgcs-core-1.0.8-2.jar,local:/usr/local/spark/jars/jpsgcs-pipe-1.0.6-7.jar /usr/local/spark/jars/thold-0.0.1-1.jar

Answer 2

In this way it worked easily.. instead of specifying each jar with version separately..

#!/bin/sh
# build all other dependent jars in OTHER_JARS

JARS=`find ../lib -name '*.jar'`
OTHER_JARS=""
   for eachjarinlib in $JARS ; do    
if [ "$eachjarinlib" != "APPLICATIONJARTOBEADDEDSEPERATELY.JAR" ]; then
       OTHER_JARS=$eachjarinlib,$OTHER_JARS
fi
done
echo ---final list of jars are : $OTHER_JARS
echo $CLASSPATH

spark-submit --verbose --class <yourclass>
... OTHER OPTIONS
--jars $OTHER_JARS,APPLICATIONJARTOBEADDEDSEPERATELY.JAR

• Using tr unix command also can help like the below example.

  --jars $(echo /dir_of_jars/*.jar | tr ' ' ',')