Spark SQL fails with java.lang.NoClassDefFoundError: org/codehaus/commons/compiler/UncheckedCompileException

Question

Running a Spark SQL (v2.1.0_2.11) program in Java immediately fails with the following exception, as soon as the first action is called on a dataframe:

java.lang         


        

Answer 1

My implementation requirement is Spring-boot + Scala + Spark(2.4.5)

For this issue, solution is to exclude artifactID 'janino' and 'commons-compiler' which comes with 'spark-sql_2.12' version 2.4.5.
The reason being the updated version 3.1.2 for both artifactID 'janino' and 'commons-compiler' which comes with 'spark-sql_2.12' version 2.4.5.

After excluding, add version 3.0.8 for both artifactID 'janino' and 'commons-compiler' as separate dependency.

     
        org.apache.spark
        spark-sql_2.12
        2.4.5
        
            
                janino
                org.codehaus.janino
            
            
                commons-compiler
                org.codehaus.janino
            
        
    
    
        janino
        org.codehaus.janino
        3.0.8
    
    
        commons-compiler
        org.codehaus.janino
        3.0.8
    
    ...............
    ...............
    ...............
    ...............
    ...............