Spark submit jobs fails with no error in local mode

倖福魔咒の 提交于 2019-12-11 19:08:38

问题


I have a spark application that I am trying to execute on windows/Unix using below command

spark-submit --class com.myorg.dataquality.DataVerification  --master local[*] C:\Users\workspace\dataQuality\target\data-quality-framework-0.0.1-SNAPSHOT.jar

The job is getting terminated immediately after submission. However it is working perfect in eclipse.

C:\Users>spark-submit --class com.myorg.dataquality.DataVerification  --master local[*] C:\Users\workspace\dataQuality\target\data-quality-framework-0.0.1-SNAPSHOT.jar
        19/09/04 19:33:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        log4j:WARN No appenders could be found for logger (org.apache.spark.deploy.SparkSubmit$$anon$2).
        log4j:WARN Please initialize the log4j system properly.
        log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
C:\Users>

What could be the issue with it:

Spark Version 2.4.4
Scala Version 2.11.11

I am able to run the job using eclipse, however i am unable to execute it on windows/Linux standalone cluster. If I try spark-shell it is initiating properly.

来源:https://stackoverflow.com/questions/57790353/spark-submit-jobs-fails-with-no-error-in-local-mode

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!