“kubectl exec” results in “error: unable to upgrade connection: Unauthorized”

Question

I tried kubectl exec on a k8s 1.6.4 RBAC-enabled cluster and the error returned was: error: unable to upgrade connection: Unauthorized. docke

Answer 1

In my case (while learning Kubernetes The Hard Way, I had to configure RBAC permissions to allow the Kubernetes API Server to access the Kubelet API on each worker node. I had created a ClusterRole and ClusterRoleBinding to access the Kubelet API

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver

References:

• https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/13-kube-apiserver-to-kubelet.md

Answer 2

Run across this one on my minikube cluster version 1.12.3
If you're running a minikube cluster, upgrade minikube and it'll be fixed.

check your minikube version:

$ minikube update-check
CurrentVersion: v1.12.3
LatestVersion: v1.13.0

Minikube docs
Upgrade(Mac OS):

brew upgrade minikube

Answer 3

This is an RTFM moment... The solution was basically to follow all the steps on this page for authn, authz, or both.

I had omitted --kubelet-client-certificate and --kubelet-client-key which resulted in the error. Without these flags, kube-apiserver will fail to authenticate with kubelet when you do a kubectl exec.

My original attempt to configure authn was by reading the docs for the kubelet daemon (ie. not the one above). Hence the grave omission.