Certificates Lifecycle
Kamaji is responsible for creating the required certificates, such as:
- the Kubernetes API Server certificate
- the Kubernetes API Server kubelet client certificate
- the Datastore certificate
- the front proxy client certificate
- the konnectivity certificate (if enabled)
Also, the following kubeconfig
resources contain client certificates, which are created by Kamaji, such as:
admin
controller-manager
konnectivity
(if enabled)scheduler
All the certificates are created with the kubeadm
defaults, thus their validity is set to 1 year.
How to rotate certificates
All certificates can be rotated at the same time, or one by one: this is possible by annotating resources using
the well-known annotation certs.kamaji.clastix.io/rotate
.
$: kubectl get secret
NAME TYPE DATA AGE
k8s-133-admin-kubeconfig Opaque 1 12m
k8s-133-api-server-certificate Opaque 2 12m
k8s-133-api-server-kubelet-client-certificate Opaque 2 3h45m
k8s-133-ca Opaque 4 3h45m
k8s-133-controller-manager-kubeconfig Opaque 1 3h45m
k8s-133-datastore-certificate Opaque 3 3h45m
k8s-133-datastore-config Opaque 4 3h45m
k8s-133-front-proxy-ca-certificate Opaque 2 3h45m
k8s-133-front-proxy-client-certificate Opaque 2 3h45m
k8s-133-konnectivity-certificate kubernetes.io/tls 2 3h45m
k8s-133-konnectivity-kubeconfig Opaque 1 3h45m
k8s-133-sa-certificate Opaque 2 3h45m
k8s-133-scheduler-kubeconfig Opaque 1 3h45m
Once this operation is performed, Kamaji will trigger a certificate renewal,
reporting the rotation date time as the annotation certs.kamaji.clastix.io/rotate
value.
$: kubectl annotate secret -l kamaji.clastix.io/certificate_lifecycle_controller=x509 certs.kamaji.clastix.io/rotate=""
secret/k8s-133-api-server-certificate annotated
secret/k8s-133-api-server-kubelet-client-certificate annotated
secret/k8s-133-datastore-certificate annotated
secret/k8s-133-front-proxy-client-certificate annotated
secret/k8s-133-konnectivity-certificate annotated
$: kubectl get secrets -l kamaji.clastix.io/certificate_lifecycle_controller=x509 -ojson | jq -r '.items[] | "\(.metadata.name) rotated at \(.metadata.annotations["certs.kamaji.clastix.io/rotate"])"'
k8s-133-api-server-certificate rotated at 2025-07-15 15:15:08.842191367 +0200 CEST m=+325.785000014
k8s-133-api-server-kubelet-client-certificate rotated at 2025-07-15 15:15:10.468139865 +0200 CEST m=+327.410948506
k8s-133-datastore-certificate rotated at 2025-07-15 15:15:15.454468752 +0200 CEST m=+332.397277417
k8s-133-front-proxy-client-certificate rotated at 2025-07-15 15:15:13.279920467 +0200 CEST m=+330.222729097
k8s-133-konnectivity-certificate rotated at 2025-07-15 15:15:17.361431671 +0200 CEST m=+334.304240277
You can notice the secrets have been automatically created back, as well as a TenantControlPlane rollout with the updated certificates.
$: kubectl get pods
NAME READY STATUS RESTARTS AGE
k8s-133-67bf496c8c-27bmp 4/4 Running 0 4m52s
k8s-133-67bf496c8c-x4t76 4/4 Running 0 4m52s
The same occurs with the kubeconfig
ones.
$: kubectl annotate secret -l kamaji.clastix.io/certificate_lifecycle_controller=kubeconfig certs.kamaji.clastix.io/rotate=""
secret/k8s-133-admin-kubeconfig annotated
secret/k8s-133-controller-manager-kubeconfig annotated
secret/k8s-133-konnectivity-kubeconfig annotated
secret/k8s-133-scheduler-kubeconfig annotated
$: kubectl get secrets -l kamaji.clastix.io/certificate_lifecycle_controller=kubeconfig -ojson | jq -r '.items[] | "\(.metadata.name) rotated at \(.metadata.annotations["certs.kamaji.clastix.io/rotate"])"'
k8s-133-admin-kubeconfig rotated at 2025-07-15 15:20:41.688181782 +0200 CEST m=+658.630990441
k8s-133-controller-manager-kubeconfig rotated at 2025-07-15 15:20:42.712211056 +0200 CEST m=+659.655019677
k8s-133-konnectivity-kubeconfig rotated at 2025-07-15 15:20:46.405567865 +0200 CEST m=+663.348376504
k8s-133-scheduler-kubeconfig rotated at 2025-07-15 15:20:46.333718563 +0200 CEST m=+663.276527216
Automatic certificates rotation
The Kamaji operator will run a controller which processes all the Secrets to determine their expiration, both for the kubeconfig
, as well as for the certificates.
The controller, named CertificateLifecycle
, will extract the certificates from the Secret objects notifying the TenantControlPlaneReconciler
controller which will start a new certificate rotation.
By default, the rotation will occur the day before their expiration.
This rotation deadline can be dynamically configured using the Kamaji CLI flag --certificate-expiration-deadline
using the Go Duration syntax:
e.g.: set the value 7d
to trigger the renewal a week before the effective expiration date.
Other Datastore Drivers
Kamaji is responsible for creating the etcd
client certificate, and the generation of a new one will occur.
For other Datastore drivers, such as MySQL, PostgreSQL, or NATS, the referenced Secret will always be deleted by the Controller to trigger the rotation: the PKI management, since it's offloaded externally, must provide the renewed certificates.
Certificate Authority rotation
Kamaji is also taking care of your Tenant Clusters Certificate Authority.
This can be rotated manually like other certificates by using the annotation certs.kamaji.clastix.io/rotate
$: kubectl annotate secret k8s-133-ca certs.kamaji.clastix.io/rotate=""
secret/k8s-133-ca annotated
Once this occurs the TenantControlPlane will enter in the CertificateAuthorityRotating
status.
$: kubectl get tcp -w
NAME VERSION STATUS CONTROL-PLANE ENDPOINT KUBECONFIG DATASTORE AGE
k8s-133 v1.33.0 Ready 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 CertificateAuthorityRotating 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 Ready 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 Ready 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 Ready 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 Ready 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
k8s-133 v1.33.0 Ready 172.18.255.200:6443 k8s-133-admin-kubeconfig default 3h58m
This operation is intended to be performed manually since a new Certificate Authority requires the restart of all the components, as well as of the nodes: in such a case, you will need to distribute the new Certificate Authority and the new nodes certificates.
Given the sensibility of such operation, the Secret
controller will not check the CA, which is offering validity of 10 years as kubeadm
default values.