Airflow Relative Importing Outside /dag Directory

Question

I haven\'t been able to move common code outside of the dag directory that airflow uses. I\'ve looked in the airflow source and found imp.load_source.

Is it possible to

Answer 1

Just add __init__.py files in all 3 folders. it should work. Infact every folder in my folder structure having __init__.py. I could run the code and see output.

Example folder structure could be as:

── airflow_home
     ├── __init__.py
     |──── dags
     │   ├── __init__.py
     │   ├── dag_1.py
     │   └── dag_2.py
     ├── common
         ├── __init__.py
         ├── foo.py
         └── bar.py

and dag_1.py code can be as:

from stackoverflow.src.questions.airflow_home.common.bar import bar_test

def main():
    bar_test()

main()

This peace of code i am running from my pycharm. Your airflow_home's folder path in my pycharm is stackoverflow/src/questions/airflow_home/

And bar.py code is

def bar_test():
    print "bar hara"

Answer 2

Other way instead of adding __init__.py file is to add the following include at the top of dag script:

import sys
import os
sys.path.insert(0,os.path.abspath(os.path.dirname(__file__)))

Answer 3

Add your airflow home path to PYTHONPATH

export AIRFLOW_HOME=/usr/local/airflow
export PYTHONPATH="${PYTHONPATH}:${AIRFLOW_HOME}"

Dockerfile

ENV AIRFLOW_HOME=/usr/local/airflow
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"