Avoiding Error 429 (quota exceeded) while working with Google Cloud Registry

北城以北 提交于 2020-06-17 15:51:26

问题


I'm hitting 429 error in Google Container Registry, too many images are pulled simultaneously

Error: Status 429 trying to pull repository [...] "Quota Exceeded."

There is a Kubernetes cluster with multiple nodes and pods implement Kubeflow steps. In the Google guide they suggest the following:

To avoid hitting the fixed quota limit, you can:
- Increase the number of IP addresses talking to Container Registry. 
  Quotas are per IP address.
- Add retries that introduce a delay. 
  For example, you could use exponential backoff.

Questions:

  • How to increase the number of IP addresses talking to Container Registry?
  • How and where is it possible to add retries that introduce a delay? Pods are run as steps in the Kubeflow pipelines.

回答1:


It seems nothing can be done in terms of the Cloud Registry quota limits because those are fixed. According to Container Registry > Doc > Resources > Quotas and limits:

Any request sent to Container Registry has a 2 hour timeout limit.

The fixed rate limits per client IP address are:

  • 50,000 HTTP requests every 10 minutes
  • 1,000,000 HTTP requests per day

Google provides support for GKE, but Kubeflow itself is not supported by Google. This question should be addressed to the Kubeflow support.

A Kubeflow issue with breaching quota limits and a question about how to get container pulls to use more IP addresses can be registered on the project page on GitHub: https://github.com/kubeflow/kubeflow/issues

Other support options are available here: https://www.kubeflow.org/docs/other-guides/support/

If you use CLI, you may try to customize Kubeflow configuration file before deployment or split it into separate deployments to overcome Cloud Registry quota limits. This approach helps for some complicated deployments. An important thing here is to take care of dependencies. First run

kfctl build -v -f ${CONFIG_URI}

make changes in the file ${KF_DIR}/kfctl_gcp_iap.yaml, and then run

kfctl apply -v -f ${CONFIG_URI}


来源:https://stackoverflow.com/questions/62141490/avoiding-error-429-quota-exceeded-while-working-with-google-cloud-registry

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!