Writing and importing custom plugins in Airflow

Question

This is actually two questions combined into one.

My AIRFLOW_HOME is structured like

airflow
+-- dags
+-- plugins
    +-- __init__.py
    +         


        

Answer 1

I figured this out by doing some trial and error. This is the final structure of my AIRFLOW_HOME folder

airflow 
+-- dags 
+-- plugins
    +-- __init__.py
    +-- plugin_name.py
    +-- hooks
        +-- __init__.py
        +-- my_hook.py 
        +-- another_hook.py 
    +-- operators
        +-- __init__.py
        +-- my_operator.py 
        +-- another_operator.py 
    +-- sensors 
    +-- utils

In plugin_name.py, I extend the AirflowPlugin class

# plugin_name.py

from airflow.plugins_manager import AirflowPlugin
from hooks.my_hook import *
from operators.my_operator import *
from utils.my_utils import *
# etc

class PluginName(AirflowPlugin):

    name = 'plugin_name'

    hooks = [MyHook]
    operators = [MyOperator]
    macros = [my_util_func]

In my custom operators which use my custom hooks, I import them like

# my_operator.py

from hooks.my_hook import MyHook

Then in my DAG files, I can do

# sample_dag.py

from airflow.operators.plugin_name import MyOperator

It is necessary to restart the webserver and scheduler. Took me a while to figure out.

This also facilitates testing since the imports within the custom classes are relative to the sub modules within the folder plugins. I wonder if I can omit the __init__.py file inside plugins, but since everything is working I didn't try doing that.