airflow

问题 I need access the parameter passed by BigqueryOperator in sql file, but I am getting error ERROR - queryParameters argument must have a type <class 'dict'> not <class 'list'> I am using below code: t2 = bigquery_operator.BigQueryOperator( task_id='bq_from_source_to_clean', sql='prepare.sql', use_legacy_sql=False, allow_large_results=True, query_params=[{ 'name': 'threshold_date', 'parameterType': { 'type': 'STRING' },'parameterValue': { 'value': '2020-01-01' } }], destination_dataset_table="{

问题 My airflow bash operator suddenly getting an error ModuleNotFoundError: No module named 'pandas' It was worked perfectly for the last 2 months but stop working from yesterday. I didn't change anything on the server or didn't update my server. 回答1: BashOperator simply runs bash command on your machine. So check if your machine has pandas installed on it. If not, do pip install pandas 来源： https://stackoverflow.com/questions/53953410/getting-modulenotfounderror-no-module-named-pandas-error-on

问题 I try to run a dynamic schedule_interval in Airflow like below. But it works only when I run the dag manually. Could you please help me say, if the dynamic schedule_interval could be a reason, why the dag does not run automatically or is there any other reasons for that? if datetime.today().day == 1: schedule_interval = '00 07 * * *' else: schedule_interval = '00 07 * * 1' Thank you! 回答1: You shouldn't set the schedule_interval to be dynamic like this as it can lead to unexpected results (as

问题 I try to run a dynamic schedule_interval in Airflow like below. But it works only when I run the dag manually. Could you please help me say, if the dynamic schedule_interval could be a reason, why the dag does not run automatically or is there any other reasons for that? if datetime.today().day == 1: schedule_interval = '00 07 * * *' else: schedule_interval = '00 07 * * 1' Thank you! 回答1: You shouldn't set the schedule_interval to be dynamic like this as it can lead to unexpected results (as

问题 Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh? 回答1: You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work

问题 Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh? 回答1: You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work

问题 Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh? 回答1: You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work

问题 Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh? 回答1: You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work

问题 Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh? 回答1: You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work

问题 Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh? 回答1: You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work