How to wait for an asynchronous event in a task of a DAG in a workflow implemented using Airflow?

Question

My workflow implemented using Airflow contains tasks A, B, C, and D. I want the workflow to wait at task C for an event. In Airflow sensors are used to check for some condit

Answer 1

One possible solution is using a sensor that waits forever until something external manually sets its state to success.

So you'd have some sort of dummy sensor:

class DummySensor(BaseSensorOperator):

    def poke(self, context):
        return False

Initialized like this:

task_c = DummySensor(
    dag=dag,
    task_id='task_c',
    interval=600,  # something fairly high since we're not polling for anything, just to check when to timeout
    timeout=3600,  # if nothing externally sets state to success in 1 hour, task will fail so task D does not run
)

When Task C starts, it will just sit in RUNNING state. Then you can use the REST API Plugin to set Task C's state to SUCCESS when the condition is met. Task D and other downstream tasks will then get kicked off.

The downside to this is the dummy sensor still holds onto a worker slot while it waits doing nothing.