问题
I'm hitting 429 error in Google Container Registry, too many images are pulled simultaneously
Error: Status 429 trying to pull repository [...] "Quota Exceeded."
There is a Kubernetes cluster with multiple nodes and pods implement Kubeflow steps. In the Google guide they suggest the following:
To avoid hitting the fixed quota limit, you can:
- Increase the number of IP addresses talking to Container Registry.
Quotas are per IP address.
- Add retries that introduce a delay.
For example, you could use exponential backoff.
Questions:
- How to increase the number of IP addresses talking to Container Registry?
- How and where is it possible to add retries that introduce a delay? Pods are run as steps in the Kubeflow pipelines.
回答1:
It seems nothing can be done in terms of the Cloud Registry quota limits because those are fixed. According to Container Registry > Doc > Resources > Quotas and limits:
Any request sent to Container Registry has a 2 hour timeout limit.
The fixed rate limits per client IP address are:
- 50,000 HTTP requests every 10 minutes
- 1,000,000 HTTP requests per day
Google provides support for GKE, but Kubeflow itself is not supported by Google. This question should be addressed to the Kubeflow support.
A Kubeflow issue with breaching quota limits and a question about how to get container pulls to use more IP addresses can be registered on the project page on GitHub: https://github.com/kubeflow/kubeflow/issues
Other support options are available here: https://www.kubeflow.org/docs/other-guides/support/
If you use CLI, you may try to customize Kubeflow configuration file before deployment or split it into separate deployments to overcome Cloud Registry quota limits. This approach helps for some complicated deployments. An important thing here is to take care of dependencies. First run
kfctl build -v -f ${CONFIG_URI}
make changes in the file ${KF_DIR}/kfctl_gcp_iap.yaml, and then run
kfctl apply -v -f ${CONFIG_URI}
来源：https://stackoverflow.com/questions/62141490/avoiding-error-429-quota-exceeded-while-working-with-google-cloud-registry