Airflow using template files for PythonOperator

Question

The method of getting a BashOperator or SqlOperator to pick up an external file for its template is somewhat clearly documented, but looking at the Pyt

Answer 1

Recently I came across the same issue and finally solved it. @Ardan 's solution is correct but just want to repeat with a more complete answer with some details in how Airflow works for the newcomers.

Of course you first need one of this:

from airflow.operators.python_operator import PythonOperator

class SQLTemplatedPythonOperator(PythonOperator):

    # somehow ('.sql',) doesn't work but tuple of two works...
    template_ext = ('.sql','.abcdefg')

Assuming you have a sql template file like below:

# stored at path: $AIRFLOW_HOME/sql/some.sql
select {{some_params}} from my_table;

First make sure you add your folder to the search path in your dag params.

Do not pass template_searchpath to args and then pass args to DAG!!!! It doesn't work.

dag = DAG(
    dag_id= "some_name",
    default_args=args,
    schedule_interval="@once",
    template_searchpath='/Users/your_name/some_path/airflow_home/sql'
)

Then your operator call will be

SQLTemplatedPythonOperator(
        templates_dict={'query': 'some.sql'},
        op_kwargs={"args_directly_passed_to_your_function": "some_value"},
        task_id='dummy',
        params={"some_params":"some_value"},
        python_callable=your_func,
        provide_context=True,
        dag=dag,
    )

Your function will be:

def your_func(args_directly_passed_to_your_function=None):
    query = context['templates_dict']['query']
    dome_some_thing(query)

Some explanations:

1. Airflow uses values from the context to render your template. To manually add it to the context, you can use the params field like above.

2. PythonOperator does not take template file extension from the template_ext field any more like @Ardan mentioned. The source code is here. It only takes extension from self.__class__.template_ext.

3. Airflow loops through the template_dict field and if value.endswith(file_extension) == True, then it renders the template.