问题
I'm using Kubernetes v1.8.14 on custom built CoreOS cluster:
$ kubectl version --short
Client Version: v1.10.5
Server Version: v1.8.14+coreos.0
When trying to create the following ClusterRole:
$ cat ClusterRole.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
I get the following error:
$ kubectl create -f ClusterRole.yml
Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{cluster-admin [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
As far as I can tell I'm connecting as cluster-admin, therefore should have sufficient permissions for what I'm trying to achieve. Below are relevant cluster-admin config:
$ cat ~/.kube/config
apiVersion: v1
kind: Config
current-context: dev
preferences:
colors: true
clusters:
- cluster:
certificate-authority: cluster-ca.pem
server: https://k8s.loc:4430
name: dev
contexts:
- context:
cluster: dev
namespace: kube-system
user: cluster-admin
name: dev
users:
- name: cluster-admin
user:
client-certificate: cluster.pem
client-key: cluster-key.pem
$ kubectl get clusterrole cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-30T14:44:44Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "1164791"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
uid: 196ffecc-9407-11e8-bd67-525400ac0b7d
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
$ kubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-30T14:44:45Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "1164832"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
uid: 19e516a6-9407-11e8-bd67-525400ac0b7d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters
$ kubectl get serviceaccount cluster-admin -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: 2018-07-30T13:32:13Z
name: cluster-admin
namespace: kube-system
resourceVersion: "1158783"
selfLink: /api/v1/namespaces/kube-system/serviceaccounts/cluster-admin
uid: f809e079-93fc-11e8-8b85-525400546bcd
secrets:
- name: cluster-admin-token-t7s4c
I understand this is RBAC problem, but have no idea how further debug this.
Edit-1.
I tried the suggested, no joy unfortunately...
$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-07-31T09:21:34Z
name: cluster-admin-binding
resourceVersion: "1252260"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
uid: 1e1c0647-94a3-11e8-9f9b-525400ac0b7d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cluster-admin
namespace: default
$ kubectl describe secret $(kubectl get secret | awk '/cluster-admin/{print $1}')
Name: cluster-admin-token-t7s4c
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=cluster-admin
kubernetes.io/service-account.uid=f809e079-93fc-11e8-8b85-525400546bcd
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1785 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
Then amended ~/.kube/config:
$ cat ~/.kube/config
apiVersion: v1
kind: Config
current-context: dev
preferences:
colors: true
clusters:
- cluster:
certificate-authority: cluster-ca.pem
server: https://k8s.loc:4430
name: dev
contexts:
- context:
cluster: dev
namespace: kube-system
user: cluster-admin-2
name: dev
users:
- name: cluster-admin
user:
client-certificate: cluster.pem
client-key: cluster-key.pem
- name: cluster-admin-2
user:
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
And then tried to apply the same ClusterRole, which rendered the same error:
$ kubectl apply -f ClusterRole.yml
Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&{system:serviceaccount:kube-system:cluster-admin f809e079-93fc-11e8-8b85-525400546bcd [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
Below are the flags which I use to start apiserver:
containers:
- name: kube-apiserver
image: quay.io/coreos/hyperkube:${K8S_VER}
command:
- /hyperkube
- apiserver
- --bind-address=0.0.0.0
- --etcd-servers=${ETCD_ENDPOINTS}
- --allow-privileged=true
- --service-cluster-ip-range=${SERVICE_IP_RANGE}
- --secure-port=443
- --advertise-address=${ADVERTISE_IP}
- --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
- --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
- --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --client-ca-file=/etc/kubernetes/ssl/ca.pem
- --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --runtime-config=extensions/v1beta1/networkpolicies=true
- --anonymous-auth=false
- --authorization-mode=AlwaysAllow,RBAC,Node
And here are the scripts, which I use to generate my tls certs:
root ca:
openssl genrsa -out ca-key.pem 4096
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=kube-ca"
apiserver:
cat > openssl.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = ${MASTER_LB_DNS}
IP.1 = ${K8S_SERVICE_IP}
IP.2 = ${MASTER_HOST}
EOF
openssl genrsa -out apiserver-key.pem 4096
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 3650 -extensions v3_req -extfile openssl.cnf
cluster-admin:
openssl genrsa -out cluster-admin-key.pem 4096
openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin"
openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
I hope this gives you more insight, what's wrong with my system.
Edit-2.
I noted a slight discrepancy between my system configuration and what @MarcinRomaszewicz suggested, thus the namespace of the cluster-admin ServiceAccount, in my case it is in the kube-system as opposed to the default namespace:
$ kubectl delete clusterrolebinding cluster-admin-binding
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" deleted
$ kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin --serviceaccount=kube-system:cluster-admin
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" created
$ kubectl apply -f ClusterRole.yml
clusterrole.rbac.authorization.k8s.io "system:coredns" created
However it still doesn't work with my certificates...
Edit-3.
As suggested in the comments, in order for apiserver to recognize the user as the cluster-admin, Subject line in the certificate of that user must contain the following items: Subject: CN = cluster-admin, O = system:masters. One way to generate such a certificate is as follows:
openssl genrsa -out cluster-admin-key.pem 4096
openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin/O=system:masters"
openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
回答1:
There isn't enough information here to answer your question.
It sounds like you are running into privilege escalation prevention: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping
This would mean you aren't actually running as cluster-admin. Check your kubectl config. You might be running as "admin" constrained to a particular name space, for example.
(edit based on comment below)
Your identity to k8s is established by the contents of your cluster.pem certificate, not the user name from kubeconfig, since that user name is only valid inside the kubeconfig file. Your actual user is determined by that certificate.
I see that you have a service account named cluster-admin, but it is not a member of "system:masters", since groups are something that are a property of the authentication system which authenticates users - you need to create an explicit cluster role binding to bind your cluster-admin service account to the cluster-admin clusterrole.
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --serviceaccount=default:cluster-admin
You should see the clusterrole now bound with your service account.
$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-07-30T22:02:33Z
name: cluster-admin-binding
resourceVersion: "71152"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
uid: 42a2862c-9444-11e8-8b71-080027de17da
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cluster-admin
namespace: default
Note at the bottom, that the binding applies to "ServiceAccount", not group.
Your service account has an access token, use that to authenticate instead of your certificate. I made myself a cluster-admin service account, and this is how I get the token:
$ kubectl describe secret $(kubectl get secret | grep cluster-admin | awk '{print $1}')
Name: cluster-admin-token-96vdz
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name=cluster-admin
kubernetes.io/service-account.uid=f872f08b-9442-11e8-8b71-080027de17da
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItYWRtaW4tdG9rZW4tOTZ2ZHoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY4NzJmMDhiLTk0NDItMTFlOC04YjcxLTA4MDAyN2RlMTdkYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmNsdXN0ZXItYWRtaW4ifQ.<signature snipped>
ca.crt: 1066 bytes
namespace: 7 bytes
Update kubeconfig to authenticate yourself using that token, instead of the certificate you are currently using, and you should be successfully authenticated as that cluster-admin service account.
(edit 2) It turned out that the certificate being used to authenticate into Kubernetes did not have any identity claims about the user. Kubernetes relies on authentication modules to authenticate users, in this case, based on certificates. It was expecting the certificate to contain a claim which put the user into the "system:masters" group, by setting the Organization to "system:masters".
There are many moving pieces here. The problem had nothing to do with service accounts or roles, but rather in user authentication, which is very opaque.
来源：https://stackoverflow.com/questions/51598467/kubernetes-rbac-forbidden-attempt-to-grant-extra-privileges