Spark 1.6.Token can be issued only with kerberos or web authentication

我是研究僧i 提交于 2020-01-24 09:35:07

问题


I call kinit keytab right before spark-submit in my shell driver script. The thing is, its working by itself, but when I call the shell driver scrip it through Oozie, I got this error:

Stdoutput py4j.protocol.Py4JJavaError: An error occurred while calling 
o49.saveAsTextFile.
Stdoutput : org.apache.hadoop.ipc.RemoteException(java.io.IOException): 
Delegation Token can be issued only with kerberos or web authentication

The issue is probably here

file.coalesce(1,True).saveAsTextFile(FQDNofHadoop+output) 

EDIT: In my script is: kinit -k -t /home/me/me.keytab me@DOMAIN.HAD

EDIT: Working solution:

I used

spark-submit --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \ 

and execution of pyspark script in oozie worked with no error (even writing into hive table and so on). Log4j logger didn't work (with standalone scripts it does), but at least print() (stdout into yarn logs) yes ...

Thx


回答1:


I used --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \ and execution of pyspark script worked with no error (even writing into hive table and so on). Logger didn't work, but at least print() yes ...



来源:https://stackoverflow.com/questions/45376108/spark-1-6-token-can-be-issued-only-with-kerberos-or-web-authentication

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!