How to clean up after a GKE cluster created with gcloud container clusters create?

问题

I'm creating Kubernetes clusters programmatically for end-to-end tests in GitLab CI/CD. I'm using gcloud container clusters create. I'm doing this for half a year and created and deleted a few hundred clusters. The cost went up and down. Now, I got an unusually high bill from Google and I checked the cost breakdown. I noticed that the cost is >95% for "Storage PD Capacity". I found out that gcloud container clusters delete never deleted the Google Compute Disks created for Persistent Volume Claims in the Kubernetes cluster.

How can I delete those programmatically? What else could be left running after deleting the Kubernetes cluster and the disks?

回答1:

Suggestions:

1. To answer your immediate question: you can programatically delete your disk resource(s) with the Method: disks.delete API.

2. To determine what other resources might have been allocated, look here: Listing all Resources in your Hierarchy.

3. Finally, this link might also help: GKE: Understanding cluster resource usage

回答2:

Because this part of the answer is lengthy:

gcloud compute disks create disk-a \
--size=10gb \
--zone=us-west1-a \
--labels=something=monday \
--project=${PROJECT}

gcloud compute disks create disk-b \
--size=10gb \
--zone=us-west1-b \
--labels=something=else \
--project=${PROJECT}

Then:

ID=$(gcloud compute disks list \
--filter="name~disk zone~us-west1 labels.something=else" \
--format="value(id)" \
--project=${PROJECT}) && echo ${ID}

NB

• the filter AND is implicit and omitted
• you may remove terms as needed
• you should make the filter as specific as possible

And -- when you're certain as deletion is irrecoverable:

gcloud compute disks delete ${ID} --project=${PROJECT} --region=${REGION}

If there are multiple matches, you can iterate:

IDS=$(gcloud compute disks list ...)
for ID in ${IDS}
do
  gcloud compute disks delete ${ID}
done

If you prefer -- the awesome jq, you'll have a general-purpose way (not gcloud-specific):

gcloud compute disks list \
--project=${PROJECT} \
--format=json \
| jq --raw-output '.[] | select(.name | contains("disk")) | select(.zone | contains("us-west1")) | select(.labels.something=="else")'
...

来源：https://stackoverflow.com/questions/59129958/how-to-clean-up-after-a-gke-cluster-created-with-gcloud-container-clusters-creat