Livy REST Spark java.io.FileNotFoundException:

时光总嘲笑我的痴心妄想 提交于 2019-12-13 03:58:27

问题


I am newer in BigData, i have tried to call spark jobs with apache Livy . With submit command line works fine . with livy i have exception

  • the command line :

    curl -X POST --data '{"file": "/user/romain/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi"}' -H 'Content-Type: application/json' http://localhost:8998/batches

  • Livy logs :

2019-06-01 00:43:19,160 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.io.FileNotFoundException: File hdfs://localhost:9000/home/spark-2.4.3-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.3.jar does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795) .......org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


回答1:


related to @cricket_007 comment resolved by execute hadoop fs -copyFromLocal command line




回答2:


If it is in local machine then try using 'file':'local:<path/to/file>



来源:https://stackoverflow.com/questions/56414713/livy-rest-spark-java-io-filenotfoundexception

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!