Execute the scala script through spark-shell in silent mode

大憨熊 提交于 2020-02-28 13:52:15

问题


Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala", after the execution, I am getting into the scala interactive mode. I don't want to get into there.

I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.

spark-shell -i "file.scala"

after execution, I get into

scala>

I don't want to get into the scala> mode

Updating (October 2019) for a script that terminates

This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script).
See this question with a good example.

It also needs a "silent mode", it is expected to not pollute the output.txt.

Suppose Spark v2.2+.


PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler... Please, "let's compile!" is not an answer here.


回答1:


spark-shell -i file.scala keeps the interpreter open in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.

If logging is requiered you can use something like this:

spark-shell < file.scala > test.log 2>&1 &

If you have limitations on editing file and you can't add System.exit(0), use:

echo :quit | scala-shell -i file.scala

UPD

If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here. Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.




回答2:


The best way is definitively to compile your scala code to a jar and use spark-submit but if you're simply looking for a quick iteration loop, you can simply issue a :quit after parsing your scala code:

echo :quit | scala-shell -i yourfile.scala



回答3:


Adding onto @rluta's answer. You can place the call to spark-shell command inside a shell script. Say the below in a shell script:

spark-shell < yourfile.scala

But this would require you to keep the lines of code within a line in case a statement is written on different lines.

OR

echo :quit | spark-shell -i yourfile.scala

This should



来源:https://stackoverflow.com/questions/57686781/execute-the-scala-script-through-spark-shell-in-silent-mode

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!