问题
I want to move files in S3 using AWS oozie. I want to run
aws s3 mv s3://temp/*.zip s3://temp/processed_files/. --recursive
How I can do this in oozie?
EDIT 1
2015-11-12 10:18:55,758 WARN ShellActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] Launcher exception: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:93)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:55)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:65)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:452)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 17 more
2015-11-12 10:18:55,838 INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] ERROR is considered as FAILED for SLA
2015-11-12 10:18:55,880 INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@killemail] Start action [0000118-151029144311676-oozie-oozi-W@killemail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
Screen shot of Hue Oozie screen shot and error log.
回答1:
if you did what Helmut Zechmann said and it's still not working, check that the problem is not linked to your shell script which has windows line endings instead of unix one.
try to convert it in the correct format using notepad++ or another tool.
回答2:
Write a java class to move files in s3 and implement it as a java action in oozie.
回答3:
The problem is that oozie cannot find your script. Put your script into hdfs and provide an alies for the script using the file element of the shell action:
<file>hdfs://path/to/move.sh#move.sh</file>
This creates an alias from the hdfs path to the name provided after the #, in this case move.sh.
Now you can reference move.sh in the exec element of your script.
See https://oozie.apache.org/docs/3.3.0/DG_ShellActionExtension.html for reference.
来源：https://stackoverflow.com/questions/33667660/move-files-in-s3-using-oozie