Recovering from expired Kubernetes API Server Certificates

0 Comments
Samuel MARTIN MORO

Today, it’s been over a year I had not time to focus on my Kubernetes labs, which I am now reminded about as I am no longer able to query their API. API Server certificate expired a few days ago.

Recovering from this is relatively easy. First, connecting to one of your control plane nodes, we would generate new certificates:

cd /etc/kubernetes
cp -rp ssl ssl.old
cd ssl
kubeadm certs renew apiserver
kubeadm crets renew apiserver-kubelet-client
kubeadm certs renew front-proxy-client

Next, we would re-generate the kubernetes-admin kubeconfig file:

kubeadm kubeconfig user --client-name kubernetes-admin \
    --config=/etc/kubernetes/kubeadm-config.yaml \
    --org system:masters >/etc/kubernetes/admin.conf

Then, make sure to share those new certificates and kubeconfig with your other control plane nodes:

scp -rp /etc/kubernetes/admin.conf /etc/kubernetes/ssl \
    root@masterX:/etc/kubernetes/

Make sure to restart kube-apiserver pods:

crictl ps | grep kube-apiserver
crictl stop <container-id>
crictl rm <container-id>
crictl ps | grep kube-apiserver

Once removed, a new kube-apiserver container should be starting up. That one would be using your new certificates: you should be recovering access to cluster API at that stage. Still we’re not done.

Then, we can proceed with kubespray, applying the cluster playbook, which should finish to restart components. You could otherwise reboot all nodes. Or restart kube-controller-managers & scheduler, then kubelet.

I’m a bit surprised that kubespray playbooks were unable to get that rotation working — running the cluster playbook, which usually fixes broken nodes or cluster configuration, was not helpful here. It does not seem to rotate kube-apiserver certificates (I could see tasks checking for their SAN, and then it keeps going, until it fails querying the API).
Still, we can see how simple it is to recover from a bad case of not paying attention to my own monitoring.
As usual, Kubernetes shines by its ease of use and reliability, despite my best effort to crash it!