Airflow using template files for PythonOperator

后端 未结 4 1134
遥遥无期
遥遥无期 2021-01-31 04:49

The method of getting a BashOperator or SqlOperator to pick up an external file for its template is somewhat clearly documented, but looking at the Pyt

4条回答
  •  栀梦
    栀梦 (楼主)
    2021-01-31 05:18

    As of Airflow 1.8, the way the PythonOperator replaces its template_ext field in __init__ doesn't work. Tasks only check template_ext on the __class__. To create a PythonOperator that picks up SQL template files you only need to do the following:

    class SQLTemplatedPythonOperator(PythonOperator):
        template_ext = ('.sql',)
    

    And then to access the SQL from your task when it runs:

    SQLTemplatedPythonOperator(
        templates_dict={'query': 'my_template.sql'},
        params={'my_var': 'my_value'},
        python_callable=my_func,
        provide_context=True,
    )
    
    def my_func(**context):
        context['templates_dict']['query']
    

提交回复
热议问题