Apache Airflow - trigger/schedule DAG rerun on completion (File Sensor)

Question

Good Morning.

I\'m trying to setup a DAG too

1. Watch/sense for a file to hit a network folder
2. Process the file
3. Archive the file

Answer 1

Dmitris method worked perfectly.

I also found in my reading setting schedule_interval=None and then using the TriggerDagRunOperator worked equally as well

trigger = TriggerDagRunOperator(
    task_id='trigger_dag_RBCPV99_rerun',
    trigger_dag_id="RBCPV99_v2",
    dag=dag)

sensor_task >> proccess_task >> archive_task >> trigger

Answer 2

Set schedule_interval=None and use airflow trigger_dag command from BashOperator to launch next execution at the completion of the previous one.

trigger_next = BashOperator(task_id="trigger_next", 
           bash_command="airflow trigger_dag 'your_dag_id'", dag=dag)

sensor_task >> proccess_task >> archive_task >> trigger_next

You can start your first run manually with the same airflow trigger_dag command and then trigger_next task will automatically trigger the next one. We use this in production for many months now and and it runs perfectly.