Airflow : Run a task when some upstream is skipped by shortcircuit

Question

I have a task that I\'ll call final that has multiple upstream connections. When one of the upstreams gets skipped by ShortCircuitOperator this tas

Answer 1

I'm posting another possible workaround for this since this is a method that does not require a custom operator implementation.

I was influenced by the solution in this blog using a PythonOperator which raises an AirflowSkipException which skips the task itself and then downstream tasks individually.

https://godatadriven.com/blog/the-zen-of-python-and-apache-airflow/

This then respects the trigger_rule of the final downstream task, which in my case I set to trigger_rule='none_failed'.

Modfied example as per the blog to include a final task:

def fn_short_circuit(**context):
    if <<>>:
        raise AirflowSkipException("Skip this task and individual downstream tasks while respecting trigger rules.")

check_date = PythonOperator(
    task_id="check_if_min_date",
    python_callable=_check_date,
    provide_context=True,
    dag=dag,
)

task1 = DummyOperator(task_id="task1", dag=dag)
task2 = DummyOperator(task_id="task2", dag=dag)
work = DummyOperator(dag=dag, task_id='work')
short = ShortCircuitOperator(dag=dag, task_id='short_circuit', python_callable=fn_short_circuit
final_task = DummyOperator(task_id="final_task",
    trigger_rule='none_failed',
    dag=dag)


task_1 >> short >> work >> final_task
task_1 >> task_2 >> final_task