How to solve the error when file doesn't exist in setting coordinator oozie

问题

How to solution when error file doesnt exist in setting coordinator oozie:

I have error in log coodinator:

Pig logfile dump:

Backend error message

Error: java.io.FileNotFoundException: File does not exist: /user/hdfs/jay/part-0.tmp

settingan coordinator:

<coordinator-app name="tes-ng" frequency="${coord:minutes(15)}"
start="2015-12-07T10:30+0700" end="2017-02-28T23:00+0700" timezone="Asia/Jakarta"
xmlns="uri:oozie:coordinator:0.1" xmlns:sla="uri:oozie:sla:0.1">
<controls>
    <execution>LAST_ONLY</execution>
</controls>
<datasets>
    <dataset name="INPUT_DS" frequency="${coord:minutes(15)}"
        initial-instance="2015-02-16T016:00+0700" timezone="Asia/Jakarta">
        <uri-template>${nameNode}/user/hdfs/jay/${YEAR}/${MONTH}/${DAY}/${HOUR}${MINUTE}
        </uri-template>
        <done-flag></done-flag>
    </dataset>
    <dataset name="OUTPUT_DS" frequency="${coord:minutes(15)}"
        initial-instance="2015-02-16T16:00+0700" timezone="Asia/Jakarta">
        <uri-template>${nameNode}/user/hdfs/jay/output</uri-template>
        <done-flag></done-flag>
    </dataset>
</datasets>
<input-events>
    <data-in name="INPUT" dataset="INPUT_DS">
        <instance>${coord:current(-2)}</instance>
    </data-in>
</input-events>
<output-events>
    <data-out name="OUTPUT" dataset="OUTPUT_DS">
        <instance>${coord:current(-2)}</instance>
    </data-out>
</output-events>
<action>
    <workflow>
        <app-path>${appFolder}</app-path>
        <configuration>
            <property>
                <name>INPUT</name>
                <value>${coord:dataIn('INPUT')}</value>
            </property>
            <property>
                <name>OUTPUT</name>
                <value>${coord:dataOut('OUTPUT')}</value>
            </property>
        </configuration>
    </workflow>
</action>

What I want is when I get error File does not exist, oozie can hold until file is all ready. any idea..??

Thanks.

回答1:

The ususal way to do this is to have a proper data dependency. The process that creates your input data creates a file that signales that the data is present (e.g. _SUCCESS). If you define a in your input dataset (e.g. _SUCCESS), Oozie will periodically check for existance of this file and only start the workflow when it is available.

<dataset name="INPUT_DS" frequency="${coord:minutes(15)}"
    initial-instance="2015-02-16T016:00+0700" timezone="Asia/Jakarta">
    <uri-template>${nameNode}/user/hdfs/jay/${YEAR}/${MONTH}/${DAY}/${HOUR}${MINUTE}
    </uri-template>
    <done-flag>_SUCCESS</done-flag>
</dataset>

If you cannot have such a flag, then AFAIK the only option is to write your own input data check and plug it into Oozie (I've seen someone do that for Hive partitions).

You should also double check the initial-instance value as it seems you've put an offset in there and then specified timezone=Asia/Jakarta on top of it.

来源：https://stackoverflow.com/questions/34310152/how-to-solve-the-error-when-file-doesnt-exist-in-setting-coordinator-oozie

标签

Hadoop

apache-pig

oozie-coordinator