airflow-scheduler

问题 Following the project from here, I am trying to integrate airflow kubernetes executor using NFS server as backed storage PV. I've a PV airflow-pv which is linked with NFS server. Airflow webserver and scheduler are using a PVC airflow-pvc which is bound with airflow-pv . I've placed my dag files in NFS server /var/nfs/airflow/development/<dags/logs> . I can see newly added DAGS in webserver UI aswell. However when I execute a DAG from UI, the scheduler fires a new POD for that tasks BUT the

问题 Following the project from here, I am trying to integrate airflow kubernetes executor using NFS server as backed storage PV. I've a PV airflow-pv which is linked with NFS server. Airflow webserver and scheduler are using a PVC airflow-pvc which is bound with airflow-pv . I've placed my dag files in NFS server /var/nfs/airflow/development/<dags/logs> . I can see newly added DAGS in webserver UI aswell. However when I execute a DAG from UI, the scheduler fires a new POD for that tasks BUT the

问题 Are there any best practices that are followed for deploying new dags to airflow? I saw a couple of comments on the google forum stating that the dags are saved inside a GIT repository and the same is synced periodically to the local location in the airflow cluster. Regarding this approach, I had a couple of questions Do we maintain separate dag files for separate environments? (testing. production) How to handle rollback of an ETL to an older version in case the new version has a bug? Any

问题 We are experimenting with Apache Airflow (version 1.10rc2, with python 2.7) and deploying it to kubernetes, webserver and scheduler to different pods, and the database is as well using cloud sql, but we have been facing out of memory problems with the scheduler pod. At the moment of the OOM, we were running only 4 example Dags (approximately 20 tasks). The memory for the pod is 1Gib. I've seen in other posts that a task might consume approximately 50Mib of memory when running, and all task

问题 We are experimenting with Apache Airflow (version 1.10rc2, with python 2.7) and deploying it to kubernetes, webserver and scheduler to different pods, and the database is as well using cloud sql, but we have been facing out of memory problems with the scheduler pod. At the moment of the OOM, we were running only 4 example Dags (approximately 20 tasks). The memory for the pod is 1Gib. I've seen in other posts that a task might consume approximately 50Mib of memory when running, and all task

来源： https://stackoverflow.com/questions/47112291/how-to-trigger-daily-dag-run-at-midnight-local-time-instead-of-midnight-utc-time

来源： https://stackoverflow.com/questions/63565229/schedule-airflow-job-bi-weekly

来源： https://stackoverflow.com/questions/63565229/schedule-airflow-job-bi-weekly

来源： https://stackoverflow.com/questions/57681573/how-to-fix-the-error-airflowexceptionhostname-of-job-runner-does-not-match

来源： https://stackoverflow.com/questions/61749480/dag-py-raises-airflow-exceptions-airflowexception-task-is-missing-the-start-d