gcloud app deploy - updating service default fails with code 13 Quota for instances limit exceeded, and 401 unathorizeed

问题

Created a new project, on first deploy I get this:

    Updating service [default] (this may take several minutes)...failed.                                                                                                       
ERROR: (gcloud.app.deploy) Error Response: [13] Flex operation projects/primalexchange/regions/us-east1/operations/b5a5813a-14bf-43b6-af7f-e17b15866c4f error [INTERNAL]: An internal error occurred while processing task /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2020-05-16T23:23:08.373Z4721.jc.14: Deployment Manager operation primalexchange/operation-1589671388926-5a5cc367252d6-eae4c5cb-bc5f1a6e errors: [code: "RESOURCE_ERROR"
location: "/deployments/aef-default-20200516t192110/resources/aef-default-20200516t192110"
message: "{\"ResourceType\":\"compute.beta.regionAutoscaler\",\"ResourceErrorCode\":\"403\",\"ResourceErrorMessage\":{\"code\":403,\"errors\":[{\"domain\":\"usageLimits\",\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'aef-default-20200516t192110\'. Limit: 8.0\",\"reason\":\"limitExceeded\"}],\"message\":\"Exceeded limit \'QUOTA_FOR_INSTANCES\' on resource \'aef-default-20200516t192110\'. Limit: 8.0\",\"statusMessage\":\"Forbidden\",\"requestPath\":\"https://compute.googleapis.com/compute/beta/projects/primalexchange/regions/us-east1/autoscalers\",\"httpMethod\":\"POST\"}}"

The build in the builder shows as a completed success.

It shows under versions that I have one version, with 0 instances, 100% traffic allocation, and when you click on the version it takes you to my site showing a 404 error.

Probably because it says that service is stopped, and that's probably because it didn't 'update'.

I don't think this is a 'my code' thing, 'cause I did a git checkout of a version that I once deployed under a different project successfully that basically amounts to a hello world app and while it once succeeded, it is now failing. Furthermore, I had another working deployment under the same project just a couple of days ago, but when I went to deploy with updated code, the deployment failed, with this very error, 'causing me to delete the entire project and start over, and now I'm getting the same error on this new project. How do I know it wasn't my code that caused the issue in the first place? I've had a build that failed because of my code and the logs of the build give that away. Here the build succeeds. there are no failures in the build.

So I've gone to this url https://compute.googleapis.com/compute/beta/projects/primalexchange/regions/us-east1/autoscalers which states this:

{
  "error": {
    "code": 401,
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "errors": [
      {
        "message": "Login Required.",
        "domain": "global",
        "reason": "required",
        "location": "Authorization",
        "locationType": "header"
      }
    ],
    "status": "UNAUTHENTICATED"
  }
}

and I don't know if that's my problem or something else. I don't even know what that means really. so I made sure I've got creds at 'projectid.json' and did export GOOGLE_APPLICATION_CREDENTIALS='projectid-2c8anumber842b4.json'

and i disconnected code related to my firebase login stuff.

I did these commands

gcloud services enable servicemanagement.googleapis.com
gcloud services enable servicecontrol.googleapis.com
gcloud services enable endpoints.googleapis.com

I don't know how to fix this issue. I've been trying to fix it all weekend.

I when to Quota's under IAM, and looked at the quota for autoscalers in us-east1 and I found this which is not over anything:

and the only thing with a limit of 8 is cpu's and it says I'm at 2.

I don't know what I'm messing up..

yes, billing is enabled.

I just ran across this, which maybe this will get me some place...

What's an authorization header? And why didn't I need that the first time.. or what have I messed up this time.

And I changed my code so that all you get is a misspelling of hello world and it's still failing on deploy:

import logging

from flask import Flask, jsonify, redirect, url_for
from flask import request, make_response
from flask import render_template
from image_upload import upload_blob

from google.cloud import storage
from google.cloud import datastore


from datetime import datetime
from cryptography.fernet import Fernet

from models.media import Media
from models.user import User


from google.cloud import ndb

import json

import random
from uuid import uuid4

from flask import Flask, make_response, request
from google.auth.transport import requests
import google.oauth2.id_token
# from google.cloud import firestore

from flask_wtf.csrf import CSRFProtect, generate_csrf
from werkzeug.utils import secure_filename
from werkzeug.exceptions import RequestEntityTooLarge

@app.route('/', methods=['GET'])
def hi():
    return "hellow world"


@app.errorhandler(500)
def server_error(e):
    logging.exception('An error occurred during a request.')
    return """
    An internal error occurred: <pre>{}</pre>
    See logs for full stacktrace.
    """.format(e), 500


if __name__ == '__main__':
    # This is used when running locally. Gunicorn is used to run the
    # application on Google App Engine. See entrypoint in app.yaml.

    app.run(host='127.0.0.1', port=8080, debug=True)

and now that I look at this more closely, how is this even running locally when I clearly have a double import of flask?

I will try to deploy after removing that, but i doubt that's the issue I'll be back to publish another essay about this issue shortly.

yeah that didn't matter. still failed.

This image is from the third project I started. And I got it to deploy up until now.. where it's failing with the same error:

Here's the App.yaml

runtime: python
api_version: '1'
env: flexible
threadsafe: true
handlers:
  - url: /static/(.*)
    application_readable: true
    static_files: __static__/static/\1
    require_matching_file: false
    upload: __static__/static/.*
  - url: /.*
    script: main.app
    secure: never
env_variables:
  SECRET_KEY: ____________
automatic_scaling:
  cool_down_period: 120s
  min_num_instances: 2
  max_num_instances: 20
  cpu_utilization:
    target_utilization: 0.5
network: {}
liveness_check:
  initial_delay_sec: 300
  check_interval_sec: 30
  timeout_sec: 4
  failure_threshold: 4
  success_threshold: 2
readiness_check:
  check_interval_sec: 5
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2
  app_start_timeout_sec: 300

And it looks when I deploy I'm hitting an IP address in use limit of 4 when I checked the quota, and then it goes back down to 2 when I'm not. So maybe someone can advice me of how to set my app.yaml on how to avoid this? I don't know anything about scaling stuff so I wanted to do auto scaling and this app.yaml is basically default settings.

Never mind. I had my quota increased to 6 and it still errors out.

回答1:

I'm not sure what's causing this, but I was able to resolve this same issue by setting scaling to manual or by setting max_num_instances below 8 in my app.yaml:

automatic_scaling:
  min_num_instances: 1
  max_num_instances: 7

回答2:

Google Cloud restricts resources by Quota, see:

https://cloud.google.com/docs/quota

For many billing accounts, you are limited to 8 Compute Engine instances (VMs) per project. You can request quota increases through the console:

https://console.cloud.google.com/iam-admin/quotas

App Engine Flex uses Compute Engine VMs.. There are 2 ways you may be consuming these:

• Multiple deployed versions running concurrently
• Multiple instances per deployed version.

You should be able to determine this via the App Engine dashboard and|or by listing the Compute Engine instances.

https://console.cloud.google.com/compute/instances

Please add your app.yaml to your question as this would be insightful.

You can make authenticated requests to Google Cloud services by acquiring an access token using gcloud and including this in a request header e.g.

curl \
--request GET \
--header "Authorization: Bearer $(gcloud auth print-access-token)" \
https://compute.googleapis.com/compute/beta/projects/primalexchange/regions/us-east1/autoscalers

回答3:

I got ahold of google tech support. They checked my quota. They checked my code. He ended up telling me to switch regions after I did a verbrosity=debug of deploy.

He said the problems was a lack of vm's available for google flex engine in the region I was in, and that I should either switch regions or switch to the standard google app engine environment.

And yes. I have to delete my project for the 4th time and start over to do either.

LOL

At least I know it's not something I'm doing.

UPdate: 4th project also failed. This time tech support said he'll get back to me once the engineers look at it.

note: deploys fine under standard app engine env

Update1:

"Hi,

This issue has been reported to our App Engine product team and is currently being investigated.

As of the moment , there are no ETA yet for the resolution but we will provide an update as soon as we heard any progress from the team.

Thank you."

and someone else posted a temp soln https://issuetracker.google.com/issues/157161938#comment2

I'm just leaving it under app engine standard for the time being

来源：https://stackoverflow.com/questions/61845193/gcloud-app-deploy-updating-service-default-fails-with-code-13-quota-for-instan