How to compile/package Spark 2.0 project with external jars and Maven

Question

Since version 2.0, Apache Spark is bundled with a folder \"jars\" full of .jar files. Obviously Maven will download all these jars when issuing:

mvn -e package
         


        

Answer 1

I am not sure whether I understand your problem, but I try to answer.

Based on Spark Bundling Your Application’s Dependencies documentation:

When creating assembly jars, list Spark and Hadoop as provided dependencies; these need not be bundled since they are provided by the cluster manager at runtime.

You can set scope to provided in maven pom.xml file

    org.apache.spark
    spark-core_2.11
    ${spark.version}
    
    provided

The second think I noticed is that maven build creates empty JAR.

[WARNING] JAR will be empty - no content was marked for inclusion!

If you have any other dependencies, you should package these dependencies into final jar archive file.

You can do something like below in pom.xml and run mvn package:

    
        org.apache.maven.plugins
        maven-assembly-plugin
        2.6
        
            
            
                jar-with-dependencies
            
            
              
                YOUR_MAIN_CLASS
              
            
            
            
              
                make-assembly
                package
                
                    single
                
              
            
    

Maven log should print line with building jar:

[INFO] --- maven-assembly-plugin:2.4.1:single (make-assembly) @ dateUtils ---
[INFO] Building jar: path/target/APPLICATION_NAME-jar-with-dependencies.jar

After maven packaging phase in the target folder you should see DataFetch-1.0-SNAPSHOTjar-with-dependencies.jar and you can sumbit this jar with spark-submit