kubernetes + coreos cluster - replacing certificates

Question

I have a coreos kubernetes cluster, which I started by following this article:

kubernetes coreos cluster on AWS

TLDR;

> kube-aws init
> kube-aws render
> kube-aws up

Everything worked good and I had a kubernetes coreos cluster on AWS. In the article there is a warning that said:

PRODUCTION NOTE: the TLS keys and certificates generated by kube-aws should not be used to deploy a production Kubernetes cluster. Each component certificate is only valid for 90 days, while the CA is valid for 365 days. If deploying a production Kubernetes cluster, consider establishing PKI independently of this tool first.

So I wanted to replace the default certificates, so I followed the following article:

coreos certificates

TLDR;

1. created the following self signed certificates: ca.pem, ca-key.pem
2. created the certificates for the controller: apiserver.pem, apiserver-key.pem
3. Replaced the certificates in the controller with the certificates created above, and rebooted the controller
4. created a worker certificates and replaced the certificates in the workers and rebooted them
5. configured kubectl to use the new certificates i created and also configured the context and user

Im getting a communication error between kubectl and the cluster, complaining about the certificate

Unable to connect to the server: x509: certificate signed by unknown authority

I also tried to use a signed certificate for kubectl which points to the cluster DNS, I set a DNS for the cluster.

How do I make kubectl communicate with my cluster?

Thanks in advance

EDIT:

My ~/.kube/config looks like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/Yariv/Development/workspace/bugeez/bugeez-kubernetes/credentials/ca2.pem
    server: https://kubernetes.bugeez.io
  name: bugeez
contexts:
- context:
    cluster: bugeez
    user: bugeez-admin
  name: bugeez-system
current-context: bugeez-system
kind: Config
preferences: {}
users:
- name: bugeez-admin
  user:
    client-certificate: /Users/Yariv/Development/workspace/bugeez/bugeez-kubernetes/credentials/admin2.pem
    client-key: /Users/Yariv/Development/workspace/bugeez/bugeez-kubernetes/credentials/admin-key2.pem

EDIT:

All my certificates are signed by ca2.pem, I also validated this fact by running:

openssl verify -CAfile ca2.pem <certificate-name>

EDIT:

What I think is the cause of the error is this: When I switch the keys in the controller and workers, seems like cloud-config is overwriting my new keys with the old ones. How do I replace the keys and also change cloud-config to adapt to my change?

Answer 1

An alternative solution that worked for me was to start a new cluster, and use custom certificates initially, without ever relying on the default temporary credentials.

Following the same tutorial that you used, I made the following changes:

> kube-aws init
> kube-aws render

Before kube-aws up, I created the certificates by following the tutorial. The only issue with the tutorial is that it is geared toward creating new certificates for an existing cluster. Therefore, the following changes are necessary:

• This line: $ openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca" needs to be replaced by: $ openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem

• In the openssl.cnf file, remove the lines that define the IP for the master host, and the loadbalancer, since we don't know what they will be yet. The final openssl.cnf should look something like this:

openssl.cnf

[req]
...
[req_distinguished_name]
[ v3_req ]
...
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = mydomain.net
IP.1 = ${K8S_SERVICE_IP} # 10.3.0.1
IP.2 = ${MASTER_IP} # 10.0.0.50

I also used the same worker certificate for all the worker nodes.

After the certificates are in place, enter kube-aws up.

I hope this helps you get off the ground

Answer 2

If the keys are indeed getting overwritten by your older ones, you will need to update the CloudFormation template to use the new userdata, which contains the new keys.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks.html

Answer 3

This error message basically means, that the server certificate is signed by an root ca, which the HTTP client doesn't know about. This could be caused by:

• The server certificate (apiserver.pem) wasn't signed by the ca obtained in the kubeconfig.yml (ca2.pem in your case). You can verify this with: openssl verify -CAfile ca2.pem apiserver.pem. The file apiserver.pem is the certificate passed via --tls-cert-file to the apiserver (see http://kubernetes.io/docs/admin/kube-apiserver/).
• The server certificate is self-signed. This was done in the how-to, but when the --tls-cert-file and the --tls-private-key-file flags were not set in the apiserver, then it creates self-signed ones.
• The apiserver.pem doesn't contain the ca certificate (see description of the --tls-cert-file flag in http://kubernetes.io/docs/admin/kube-apiserver/). I am not entirely sure whether the HTTPS server needs to know about the root ca, but this may also cause the problem.

Also this error message doesn't make it clear, whether this is a problem with the client certificate or the server certificate. This means the client certificate has also to be signed by a root ca.

• The client certificate ca is set by the --client-ca-file flag in the apiserver. Assuming this file is also named ca2.pem, then the client cert can be verified with: openssl verify -CAfile ca2.pem admin2.pem