Skip to content

Datastore Migration

On the Management Cluster, you can deploy one or more multi-tenant datastores as etcd, PostgreSQL, MySQL, and NATS to save the state of the Tenant Clusters. A Tenant Control Plane can be migrated from a datastore to another one without service disruption or without complex and error-prone backup & restore procedures.

This guide will assist you to live migrate Tenant's data from a datastore to another one having the same etcd driver.

Prerequisites

Assume you have a Tenant Control Plane using the default datastore:

kubectl get tcp
NAME        VERSION   STATUS   CONTROL-PLANE ENDPOINT   KUBECONFIG                   DATASTORE   AGE
tenant-00   v1.25.2   Ready    192.168.32.200:6443      tenant-00-admin-kubeconfig   default     8d

You can check a custom resource called DataStore providing a declarative description of the default datastore:

apiVersion: kamaji.clastix.io/v1alpha1
kind: DataStore
metadata:
  annotations:
  labels:
  name: default
spec:
  driver: etcd
  endpoints:
  - etcd-0.etcd.kamaji-system.svc.cluster.local:2379
  - etcd-1.etcd.kamaji-system.svc.cluster.local:2379
  - etcd-2.etcd.kamaji-system.svc.cluster.local:2379
  tlsConfig:
    certificateAuthority:
      certificate:
        secretReference:
          keyPath: ca.crt
          name: etcd-certs
          namespace: kamaji-system
      privateKey:
        secretReference:
          keyPath: ca.key
          name: etcd-certs
          namespace: kamaji-system
    clientCertificate:
      certificate:
        secretReference:
          keyPath: tls.crt
          name: etcd-root-client-certs
          namespace: kamaji-system
      privateKey:
        secretReference:
          keyPath: tls.key
          name: etcd-root-client-certs
          namespace: kamaji-system
status:
  usedBy:
  - default/tenant-00

The default datastore is installed by Kamaji Helm chart in the same namespace hosting the controller:

kubectl -n kamaji-system get pods
NAME                              READY   STATUS      RESTARTS   AGE
etcd-0                            1/1     Running     0          23d
etcd-1                            1/1     Running     0          23d
etcd-2                            1/1     Running     0          23d
kamaji-5d6cdfbbb9-bn27f           1/1     Running     0          2d19h

Install a new datastore

A managed datastore is highly recommended in production. The kamaji-etcd project provides a viable option to setup a managed multi-tenant etcd running as StatefulSet made of three replicas:

helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install dedicated clastix/kamaji-etcd -n dedicated --create-namespace --set datastore.enabled=true

You should end up with a new datastore dedicated provided by an etcd cluster:

# kubectl get datastore dedicated -o yaml
apiVersion: kamaji.clastix.io/v1alpha1
kind: DataStore
metadata:
  annotations:
  labels:
  name: dedicated
spec:
  driver: etcd
  endpoints:
  - dedicated-0.dedicated.dedicated.svc.cluster.local:2379
  - dedicated-1.dedicated.dedicated.svc.cluster.local:2379
  - dedicated-2.dedicated.dedicated.svc.cluster.local:2379
  tlsConfig:
    certificateAuthority:
      certificate:
        secretReference:
          keyPath: ca.crt
          name: dedicated-certs
          namespace: dedicated
      privateKey:
        secretReference:
          keyPath: ca.key
          name: dedicated-certs
          namespace: dedicated
    clientCertificate:
      certificate:
        secretReference:
          keyPath: tls.crt
          name: dedicated-root-client-certs
          namespace: dedicated
      privateKey:
        secretReference:
          keyPath: tls.key
          name: dedicated-root-client-certs
          namespace: dedicated
status: {}

Check the etcd cluster:

kubectl -n dedicated get sts,pods,pvc
NAME                         READY   AGE
statefulset.apps/dedicated   3/3     25h

NAME                                  READY   STATUS      RESTARTS   AGE
pod/dedicated-0                       1/1     Running     0          25h
pod/dedicated-1                       1/1     Running     0          25h
pod/dedicated-2                       1/1     Running     0          25h

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/data-dedicated-0   Bound    pvc-a5c66737-ef78-4689-b863-037f8382ed78   10Gi       RWO            local-path     25h
persistentvolumeclaim/data-dedicated-1   Bound    pvc-1e9f77eb-89f3-4256-9508-c18b71fca7ea   10Gi       RWO            local-path     25h
persistentvolumeclaim/data-dedicated-2   Bound    pvc-957c4802-1e7c-4f37-ac01-b89ad1fa9fdb   10Gi       RWO            local-path     25h

Migrate data

To migrate data from current default datastore to the new dedicated one, patch the Tenant Control Plane tenant-00 to use the new dedicated datastore:

kubectl patch --type merge tcp tenant-00 -p '{"spec": {"dataStore": "dedicated"}}'

and check the process happening in real time:

kubectl get tcp -w
NAME        VERSION   STATUS      CONTROL-PLANE ENDPOINT   KUBECONFIG                   DATASTORE   AGE
tenant-00   v1.25.2   Ready       192.168.32.200:6443      tenant-00-admin-kubeconfig   default     9d
tenant-00   v1.25.2   Migrating   192.168.32.200:6443      tenant-00-admin-kubeconfig   default     9d
tenant-00   v1.25.2   Migrating   192.168.32.200:6443      tenant-00-admin-kubeconfig   default     9d
tenant-00   v1.25.2   Migrating   192.168.32.200:6443      tenant-00-admin-kubeconfig   dedicated   9d
tenant-00   v1.25.2   Migrating   192.168.32.200:6443      tenant-00-admin-kubeconfig   dedicated   9d
tenant-00   v1.25.2   Ready       192.168.32.200:6443      tenant-00-admin-kubeconfig   dedicated   9d

During the datastore migration, the Tenant Control Plane is put in read-only mode to avoid misalignments between source and destination datastores. If tenant users try to update the data, an admission controller denies the request with the following message:

Error from server (the current Control Plane is in freezing mode due to a maintenance mode,
all the changes are blocked: removing the webhook may lead to an inconsistent state upon its completion):
admission webhook "catchall.migrate.kamaji.clastix.io" denied the request

After a while, depending on the amount of data to migrate, the Tenant Control Plane is put back in full operating mode by the Kamaji controller.

Please, note the datastore migration leaves the data on the default datastore, so you have to remove it manually.

Post migration

After migrating data to the new datastore, complete the migration procedure by restarting the kubelet.service on all the tenant worker nodes.

Troubleshooting

Migration Job Image Version

When migrating between datastores, the Kamaji controller automatically creates a migration job to transfer data from the source to the destination datastore. By default, this job uses the same image version as the running Kamaji controller. If you need to use a different image version for the migration job, you can specify it by passing extra arguments to the controller:

helm upgrade kamaji clastix/kamaji -n kamaji-system 
--set extraArgs[0]=--migrate-image=custom/kamaji:version`

Handling Private Registry Images

If the Kamaji controller images are stored in a private registry that requires authentication, the migration job will fail because it does not use any ImagePullSecret by default. You need to attach your registry secret to the kamaji-controller-manager service account, which is used by the migration job. You can do this with the following command:

kubectl -n kamaji-system patch serviceaccount kamaji-controller-manager \
        -p '{"imagePullSecrets": [{"name": "myregistry-credentials"}]}'

This command patches the kamaji-controller-manager service account to include your registry secret, allowing the migration job to pull images from the private registry successfully.