GKE Cluster can't pull (ErrImagePull) from GCR Registry in same project (GitLab Kubernetes Integration): Why?

Question

So after googling a little bit (which is polluted by people having trouble with Pull Secrets) I am posting this here — and to GCP Support (will update as I hear).

I crea

Answer 1

TL;DR — Clusters created by GitLab-Ci Kubernetes Integration will not be able to pull an image from a GCR Registry in the same project as the container images — without modifying the Node(s) permissions (scopes).

By default the Cluster Nodes created by a Cluster which was itself created by GitLab-Ci's Kubernetes Integration are created with minimal permissions (scopes) to Google Cloud services.

You can see this visually from the GCP console dashboard for your cluster, scroll down to the permissions section and look for "Storage":

This essentially means that the Node(s) running within your GitLab-Ci Kubernetes integration cluster WILL NOT have the default GCR Registry (read-only) permissions necessary to pull an image from a GCR Registry.

It also means (as far as I can tell) that even if you grant a Service Account proper access to the GCR Registry it still will not work — not totally sure I set my Service Account up properly but I believe I did.

Great.

How to fix Permissions

Basically you have two options. The first one is to create a Cluster (ie. outside of GitLab Kubernetes Integration) and then re-connect your GitLab project to THAT Cluster by following the manual 'connect to an existing Cluster' directions that can be found here: https://gitlab.com/help/user/project/clusters/index#adding-an-existing-kubernetes-cluster

The second option is to modify your permissions but that is more complicated.