AWS EMR 4.0 - How can I add a custom JAR step to run shell commands

问题

I am trying to run shell commands using steps on EMR 4.0.0 and used this link for reference - http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html

But I want to know from where to put 'command-runner.jar' in 'JAR location' field http://i.stack.imgur.com/CRicz.png

I kept 'command-runner.jar' in AWS s3 and tried to load it from that location and in 'Arguments' gave s3 location of my 'example.sh' file and after adding step it failed giving this exception

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:139)
    at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:92)
    ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 8 more

回答1:

To run a shell script stored in S3, use script-runner.jar, which is located at s3://elasticmapreduce/libs/script-runner/script-runner.jar.

command-runner.jar will work for local files only.

来源：https://stackoverflow.com/questions/34151518/aws-emr-4-0-how-can-i-add-a-custom-jar-step-to-run-shell-commands