问题

I am trying to use rules_k8s for Bazel to deploy to my Kubernetes cluster.

Thus I have this cloudbuild.yaml file, which is executed by Google Cloud Build:

steps:
  - name: gcr.io/cloud-builders/bazel
    args: ['run', '//:kubernetes.apply']

(//:kubernetes is just a k8s_objects)

On my local machine running bazel run //:kubernetes.apply works fine, but although the Google Cloud Build succeeds, it logs those errors. So the configuration is not applied to my Kubernetes cluster:

Target //:kubernetes.apply up-to-date:
  bazel-bin/kubernetes.apply
INFO: Elapsed time: 29.863s, Critical Path: 0.14s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/kubernetes.apply
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 1 total action
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.

I also get a warning from the bazel cache:

DEBUG: /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/io_bazel_rules_k8s/toolchains/kubectl/kubectl_toolchain.bzl:28:9: No kubectl tool was found or built, executing run for rules_k8s targets might not work.

P.S.: I get the same errors when using //:kubernetes.create

My setup

Deployments

load("@io_bazel_rules_k8s//k8s:object.bzl", "k8s_object")
k8s_object(
  name = "k8s_deployment",
  kind = "deployment",
  cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
  template = ":ideas.deployment.yaml",
  images = {
    "gcr.io/cents-ideas/ideas:latest": ":image"
  },
)

Services

k8s_object(
  name = "k8s_service",
  kind = "service",
  cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
  template = ":ideas.service.yaml",
)

Aggregations

load("@io_bazel_rules_k8s//k8s:objects.bzl", "k8s_objects")
k8s_objects(
    name = "k8s",
    objects = [
      ":k8s_deployment",
      ":k8s_service",
    ]
)

Final composition

k8s_objects(
    name = "kubernetes",
    objects = [
        "//services/ideas:k8s",
        # ...
    ]
)

Update

I've now tried to make my own docker image with Bazel and kubectl:

FROM gcr.io/cloud-builders/bazel

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

I pushed it to GCR and changed my cloudbuild.yaml to:

steps:
  - name: eu.gcr.io/cents-ideas/bazel-kubectl
    args: ["run", "//:kubernetes.apply"]

I firstly noticed, that the step took way longer than before. However at the end it throws an error:

$ /usr/local/bin/kubectl --kubeconfig= --cluster=gke_cents-ideas_europe-west3-a_cents-ideas --context= --user= apply -f -
error: cluster "gke_cents-ideas_europe-west3-a_cents-ideas" does not exist

Here is the full log.

回答1:

As for the updated question, now you need to authenticate somehow to GKE inside the container.

First thing, I recommend installing gcloud tool to your container. Btw, as for the huge container size 1.2 GB, that's because cloud-builders/bazel is huge :)

Have a look at our example on slim bazel container version: https://github.com/aspect-development/bazel-k8s-example/blob/master/tools/Dockerfile.dazel

And here is Dockerfile for installing gcloud and kubectl, so you can grab needed parts from both files: https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile

The second thing is authenticating, after gcloud is installed it should be easy. Overall cloudbuild step should look similar to this:

- name: <link to your container>
  entrypoint: /bin/sh
  args:
  - -c
  - |
    gcloud container clusters get-credentials cents-ideas --zone europe-west3-a --project cents-ideas
    bazel run //:kubernetes.apply

回答2:

It complains that it can't find kubectl on your machine. If GKE is used, also the gcloud sdk needs to be installed.

And also check that authentication is configured for those tools: kubectl auth, GKE auth.

回答3:

The main issue is that kubectl doesn't come with the gcr.io/cloud-builders/bazel docker image. That is why it can't be found.

There are however, efforts to implement the kubectl toolchain into Bazel manually: https://github.com/bazelbuild/rules_k8s/tree/master/toolchains/kubectl#kubectl-toolchain. But as those features are currently in experimental status, they often don't work.

More info can be found in this issue: https://github.com/bazelbuild/rules_k8s/issues/512