Debugging Broken DAGs

Question

When the airflow webserver shows up errors like Broken DAG: [] , how and where can we find the full stacktrace for these excepti

Answer 1

What you want to do is access the inner logs of the webserver so that you get the full stacktrace. My Airflow server is being executed in a Docker image so I'll use Docker to fetch these logs but the idea remains.

1. docker ps
2. fetch the PID of the webserver
3. docker logs [PID]
4. read the full logs of the given Airflow webserver.

This should contain the exact information of why your DAG build failed.

Answer 2

If you want to compile and see any syntax error, you can also try python your_dag.py

Answer 3

I try below following step by step

• airflow list_dags as mentioned by @Babcool this will list stacktrace

If you are still not able to figure out issue you run task manually and see direct errors.

pre-step set you environment variables

export AIRFLOW_HOME="/airflow/project/path"
export PYTHONPATH="/airflow/project/path"

Running dag

airflow run dag_id task_id 2020-1-11

source:

• airflow doc
• stackoverflow

If still things aren't clear you can try running code line by line in python console and check exact issue (after activating your virtual environment)

For example:

(venv) shakeel@workstation:~$ python
Python 3.7.9 (default, Aug 18 2020, 06:24:24) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from my_package.my_module import MyClass
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'my_package'
>>>

Answer 4

The accepted answer works in almost all cases to validate DAGs and debug errors if any.

If you are using docker-compose to run airflow, you should do this:

docker-compose exec airflow airflow list_dags

It runs the same command inside the running container.

Answer 5

Usually I used the command airflow list_dags which print the full stacktrace for python error found in dags.

That will work with almost any airflow command as airflow parse dags folder each time you use a airflow CLI command.