google-cloud-composer

问题 I am new in python and airflow dag. I am following below link and code which is mention in answer section. How to pass dynamic arguments Airflow operator? I am facing issue to reading yaml file, In yaml file I have some configuration related arguments. configs: cluster_name: "test-cluster" project_id: "t***********" zone: "europe-west1-c" num_workers: 2 worker_machine_type: "n1-standard-1" master_machine_type: "n1-standard-1" In DAG script I have created one task which will be create cluster,

问题 We are using Google Composer for workflow orchestration, randomly we are getting An internal server error occurred while authorizing your request. Error code 28 message while opening the web UI. We don't know the cause for this issue. How to fix this? 回答1: This issue could be given for users who try to access Airflow UI from certain location. Notice that direct access to the Airflow UI is not supported in Australia, New Zealand, and India as explained here. The product team is working on the

问题 I have submitted a Hive job using Dataproc Workflow Template with the help of Airflow operator (DataprocWorkflowTemplateInstantiateInlineOperator) written in Python. Once the job is submitted some name will be assigned as jobId (example: job0-abc2def65gh12 ). Since I was not able to get jobId I tried to pass jobId as a parameter from REST API which isn't working. Can I fetch jobId or, if it's not possible, can I pass jobId as a parameter? 回答1: The JobId will be available as part of metadata

问题 I thought I could use the command: g beta composer environments run <env> --location=us-central1 clear -- <dag_id> -s 2018-05-13 -e 2018-05-14 the clear the state of the dag runs on 2018-05-13. For some reason it doesn't work. What happens is that the CLI hangs on a message like: kubeconfig entry generated for <kube node name>. What is the expected behavior of the command above? I would expect it to clear the dag run for the interval, but I might be doing something wrong. 回答1: Running clear

问题 My Cloud Composer managed Airflow got stuck for hours since I've canceled a Task Instance that was taking too long (Let's call it Task A) I've cleared all the DAG Runs and task instances, but there are a few jobs running and one job with Shutdown state (I suppose the job of Task A) (snapshot of my Jobs). Besides, it seems that the scheduler is not running since recently deleted DAGs keep appearing in the dashboard Is there a way to kill the jobs or reset the scheduler? Any idea to un-stuck

问题 I have a few file-related use cases that I'm not sure how to best accomplish using Cloud Composer. How should I best accomplish these? 1)I need to use a private key (.pem) file to access an SFTP server. Where should this file be stored and how should it be accessed? In on-prem Airflow, I would just have the file in a folder /keys/ in the same directory as /dags/. 2)I need to move files from an SFTP server to Cloud Storage. With Airflow on prem, I download these from the SFTP server to a

问题 In some of my Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler is not fully loaded. How can I increase the number of DAGs or tasks that can run concurrently? Similarly, if my installation is under high load and I want to limit how quickly my Airflow workers pull queued tasks, what can I adjust? 回答1: Here's an expanded list of configuration options that are available in Airflow v1.10.2. Some can be set on a per-DAG or per-operator basis, and may

问题 I'm using Google Cloud Composer (managed Airflow on Google Cloud Platform) with image version composer-0.5.3-airflow-1.9.0 and Python 2.7, and I'm facing a weird issue : after importing my DAGs, they are not clickable from the Web UI (and there are no buttons "Trigger DAG", "Graph view", ...), while all works perfectly when running a local Airflow. Even if non usable from the webserver on Composer, my DAGs still exist. I can list them using CLI ( list_dags ), describe them ( list_tasks ) and

问题 According to the documentation here: https://cloud.google.com/composer/docs/release-notes Composer (GCP's Airflow) is supposed to be available for Python 3 in the console. However, I am seeing no options for Python 3 in the console. 回答1: Python3 support is a beta feature in Composer, this doc describes how to enable and use beta feature in Cloud Composer. 回答2: I was confronted with the same problem and I was able to solve it. Since there is a checkbox of Enable Beta Features on the upper

问题 I have a project with Cloud Composer and Cloud SQL. I am able to connect to Cloud SQL because i edited the yaml of airflow-sqlproxy-service and added my Cloud SQL instance on cloud proxy used for the airflow-db, mapping to port 3307. The workers can connect to airflow-sqlproxy-service on port 3307 but i think the webserver can't connect to this. Do i need to add some firewall rule to map the 3307 port so the webserver or the UI can connect to airflow-sqlproxy-service? https://i.stack.imgur