Cluster Class with Kamaji

ClusterClass is a Cluster API feature that enables template-based cluster creation. When combined with Kamaji's hosted control plane architecture, ClusterClass provides a powerful pattern for standardizing Kubernetes cluster deployments across multiple infrastructure providers while maintaining consistent control plane configurations.

Experimental Feature

ClusterClass is still an experimental feature of Cluster API. As with any experimental features it should be used with caution. Read more about ClusterClass in the Cluster API documentation.

Understanding Cluster Class

ClusterClass reduces configuration boilerplate by defining reusable cluster templates. Instead of creating individual resources for each cluster, you define a ClusterClass once and create multiple clusters from it with minimal configuration.

With Kamaji, this pattern becomes even more powerful: - Shared Control Plane Templates: The same KamajiControlPlaneTemplate works across all infrastructure providers - Infrastructure Flexibility: Deploy worker nodes on vSphere, AWS, Azure, or any supported provider while maintaining consistent control planes - Simplified Management: Hosted control planes reduce the complexity of ClusterClass templates

Enabling Cluster Class

To use ClusterClass with Kamaji, you need to enable the cluster topology feature gate before initializing the management cluster:

export CLUSTER_TOPOLOGY=true
clusterctl init --control-plane kamaji --infrastructure vsphere

This will install: - Cluster API core components with ClusterClass support - Kamaji Control Plane Provider - Your chosen infrastructure provider (vSphere in this example)

Verify the installation:

kubectl get deployments -A | grep -E "capi|kamaji"

Template Architecture with Kamaji

A ClusterClass with Kamaji consists of four main components:

1. Control Plane Template (KamajiControlPlaneTemplate): Defines the hosted control plane configuration that remains consistent across infrastructure providers.

2. Infrastructure Template (VSphereClusterTemplate): Provider-specific infrastructure configuration for the cluster.

3. Bootstrap Template (KubeadmConfigTemplate): Node initialization configuration that works across providers.

4. Machine Template (VSphereMachineTemplate): Provider-specific machine configuration for worker nodes.

Here's how these components relate in a ClusterClass:

apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: kamaji-vsphere-class
spec:
  # Infrastructure provider template
  infrastructure:
    ref:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereClusterTemplate
      name: vsphere-cluster-template

  # Kamaji control plane template - reusable across providers
  controlPlane:
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
      kind: KamajiControlPlaneTemplate
      name: kamaji-control-plane-template

  # Worker configuration
  workers:
    machineDeployments:
    - class: default-worker
      template:
        bootstrap:
          ref:
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
            kind: KubeadmConfigTemplate
            name: worker-bootstrap-template
        infrastructure:
          ref:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: VSphereMachineTemplate
            name: vsphere-worker-template

The key advantage: the KamajiControlPlaneTemplate and KubeadmConfigTemplate can be shared across different infrastructure providers, while only the infrastructure-specific templates need to change.

Creating a Cluster Class

Let's create a ClusterClass for vSphere with Kamaji. First, define the shared templates:

KamajiControlPlaneTemplate

This template defines the hosted control plane configuration:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlaneTemplate
metadata:
  name: kamaji-controlplane
  namespace: capi-templates-vsphere
spec:
  template:
    spec:
      dataStoreName: "default"  # Default datastore for etcd

      network:
        serviceType: LoadBalancer
        serviceAddress: ""
        certSANs: []

      addons:
        coreDNS: {}
        kubeProxy: {}
        konnectivity: {}

      apiServer:
        extraArgs: []
        resources:
          requests: {}
      controllerManager:
        extraArgs: []
        resources:
          requests: {}
      scheduler:
        extraArgs: []
        resources:
          requests: {}

      kubelet:
        cgroupfs: systemd
        preferredAddressTypes:
        - InternalIP

      registry: "registry.k8s.io"

KubeadmConfigTemplate

This bootstrap template configures worker nodes:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: worker-bootstrap-template
spec:
  template:
    spec:

      # Configuration for kubeadm join
      joinConfiguration:
        discovery: {}
        nodeRegistration:
          criSocket: /var/run/containerd/containerd.sock
          imagePullPolicy: IfNotPresent
          name: '{{ local_hostname }}'
          kubeletExtraArgs:
            cloud-provider: external
            node-ip: "{{ ds.meta_data.local_ipv4 }}"       

      # Commands to run before kubeadm join
      preKubeadmCommands:
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "127.0.0.1 {{ ds.meta_data.hostname }}" >> /etc/hosts

      # Commands to run after kubeadm join
      postKubeadmCommands: []

      # Users to create on worker nodes
      users: []

VSphereClusterTemplate

Infrastructure-specific template for vSphere:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereClusterTemplate
metadata:
  name: vsphere
  namespace: capi-templates-vsphere
spec:
  template:
    spec:
      server: "vcenter.sample.com"  # vCenter server address
      thumbprint: ""                # vCenter certificate thumbprint

      identityRef:
        kind: VSphereClusterIdentity  
        name: "vsphere-cluster-identity"

      failureDomainSelector: {} 
      clusterModules: []

VSphereMachineTemplate

Machine template for vSphere workers:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: vsphere-vm-base
  namespace: capi-templates-vsphere
spec:
  template:
    spec:
      # Resources will be patched by ClusterClass based on variables
      # numCPUs, memoryMiB, diskGiB are dynamically set

      # Infrastructure defaults - will be patched by ClusterClass
      server: "vcenter.sample.com"
      datacenter: "datacenter"
      datastore: "datastore"
      resourcePool: "Resources"
      folder: "vm-folder"
      template: "ubuntu-2404-kube-v1.32.0"
      storagePolicyName: ""
      thumbprint: ""

      # Network configuration (IPAM by default)
      network:
        devices:
        - networkName: "k8s-network"
          dhcp4: false
          addressesFromPools:
          - apiGroup: ipam.cluster.x-k8s.io
            kind: InClusterIPPool
            name: "{{ .builtin.cluster.name }}"  # Uses cluster name

Variables and Patching in Cluster Class

ClusterClass becomes powerful through its variable system and JSON patching capabilities. This allows the same templates to be customized for different use cases without duplicating YAML.

Variable System

Variables in ClusterClass define the parameters users can customize when creating clusters. Each variable has:

• Schema Definition: OpenAPI v3 schema that validates input
• Required/Optional: Whether the variable must be provided
• Default Values: Fallback values when not specified
• Type Constraints: Data types, ranges, and enum values

Here's how variables work in practice:

Control Plane Variables:

variables:
- name: kamajiControlPlane
  required: true
  schema:
    openAPIV3Schema:
      type: object
      properties:
        dataStoreName:
          type: string
          description: "Datastore name for etcd"
          default: "default"
        network:
          type: object
          properties:
            serviceType:
              type: string
              enum: ["ClusterIP", "NodePort", "LoadBalancer"]
              default: "LoadBalancer"
            serviceAddress:
              type: string
              description: "Pre-assigned VIP address"

Machine Resource Variables:

- name: machineSpecs
  required: true
  schema:
    openAPIV3Schema:
      type: object
      properties:
        numCPUs:
          type: integer
          minimum: 2
          maximum: 64
          default: 4
        memoryMiB:
          type: integer
          minimum: 4096
          maximum: 131072
          default: 8192
        diskGiB:
          type: integer
          minimum: 40
          maximum: 2048
          default: 100

JSON Patching System

Patches apply variable values to the base templates at cluster creation time. This enables the same template to serve different configurations.

Control Plane Patching:

patches:
- name: controlPlaneConfig
  definitions:
  - selector:
      apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
      kind: KamajiControlPlaneTemplate
      matchResources:
        controlPlane: true
    jsonPatches:
    - op: replace
      path: /spec/template/spec/dataStoreName
      valueFrom:
        variable: kamajiControlPlane.dataStoreName
    - op: replace
      path: /spec/template/spec/network/serviceType
      valueFrom:
        variable: kamajiControlPlane.network.serviceType

Machine Resource Patching:

- name: machineResources
  definitions:
  - selector:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereMachineTemplate
      matchResources:
        machineDeploymentClass:
          names: ["default-worker"]
    jsonPatches:
    - op: add  # Resources are not in base template
      path: /spec/template/spec/numCPUs
      valueFrom:
        variable: machineSpecs.numCPUs
    - op: add
      path: /spec/template/spec/memoryMiB
      valueFrom:
        variable: machineSpecs.memoryMiB

Advanced Patching Patterns

Conditional Patching:

- name: optionalVIP
  definitions:
  - selector:
      apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
      kind: KamajiControlPlaneTemplate
    jsonPatches:
    - op: replace
      path: /spec/template/spec/network/serviceAddress
      valueFrom:
        variable: kamajiControlPlane.network.serviceAddress
      # Only applies if serviceAddress is not empty
      enabledIf: "{{ ne .kamajiControlPlane.network.serviceAddress \"\" }}"

Infrastructure Patching:

- name: infrastructureConfig
  definitions:
  - selector:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereMachineTemplate
    jsonPatches:
    - op: replace
      path: /spec/template/spec/datacenter
      valueFrom:
        variable: infrastructure.datacenter
    - op: replace
      path: /spec/template/spec/datastore
      valueFrom:
        variable: infrastructure.datastore
    - op: replace
      path: /spec/template/spec/template
      valueFrom:
        variable: infrastructure.vmTemplate

Complete Cluster Class with Variables

For a comprehensive example with all variables and patches configured, see the vsphere-kamaji-clusterclass.yaml template.

Creating a Cluster from Cluster Class

With the ClusterClass defined, creating a cluster becomes remarkably simple:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: my-cluster
  namespace: default
spec:
  # Network configuration defined at cluster level
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
    serviceDomain: "cluster.local"

  topology:
    class: vsphere-standard
    classNamespace: capi-templates-vsphere
    version: v1.32.0

    controlPlane:
      replicas: 2

    workers:
      machineDeployments:
      - class: default-worker
        name: worker-nodes
        replicas: 3

    variables:
    - name: kamajiControlPlane
      value:
        dataStoreName: "etcd"
        network:
          serviceType: "LoadBalancer"
          serviceAddress: ""  # Auto-assigned if empty

    - name: machineSpecs
      value:
        numCPUs: 8
        memoryMiB: 16384
        diskGiB: 60

    - name: infrastructure
      value:
        vmTemplate: "ubuntu-2404-kube-v1.32.0"
        datacenter: "K8s-TI-dtc"
        datastore: "K8s-N01td-01"
        resourcePool: "rp-kamaji-dev"
        folder: "my-cluster-vms"

    - name: networking
      value:
        networkName: "VM-K8s-TI-cpmgmt"
        nameservers: ["8.8.8.8", "1.1.1.1"]
        dhcp4: false  # Using IPAM

Create the cluster:

kubectl apply -f my-cluster.yaml

Monitor cluster creation:

clusterctl describe cluster my-cluster
kubectl get cluster,kamajicontrolplane,machinedeployment -n default

With this approach, the same KamajiControlPlaneTemplate and KubeadmConfigTemplate can be reused when creating ClusterClasses for AWS, Azure, or any other provider. Only the infrastructure-specific templates need to change.

Cross-Provider Template Reuse

One of Kamaji's key advantages with ClusterClass is template modularity across providers. Here's how to leverage this:

Shared Templates Repository

Create a namespace for shared templates:

kubectl create namespace cluster-templates

Deploy shared Kamaji and bootstrap templates once:

kubectl apply -n cluster-templates -f kamaji-controlplane-template.yaml
kubectl apply -n cluster-templates -f kubeadm-config-template.yaml

Provider-Specific Cluster Classes

For each infrastructure provider, create a ClusterClass that references the shared templates:

AWS Cluster Class

apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: kamaji-aws-class
spec:
  controlPlane:
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
      kind: KamajiControlPlaneTemplate
      name: kamaji-controlplane
      namespace: cluster-templates  # Shared template

  infrastructure:
    ref:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSClusterTemplate
      name: aws-cluster-template  # AWS-specific

  workers:
    machineDeployments:
    - class: default-worker
      template:
        bootstrap:
          ref:
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
            kind: KubeadmConfigTemplate
            name: kubeadm
            namespace: cluster-templates  # Shared template
        infrastructure:
          ref:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
            kind: AWSMachineTemplate
            name: aws-worker-template  # AWS-specific

Azure Cluster Class

apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: kamaji-azure-class
spec:
  controlPlane:
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
      kind: KamajiControlPlaneTemplate
      name: kamaji-control-plane-template
      namespace: cluster-templates  # Same shared template

  infrastructure:
    ref:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: AzureClusterTemplate
      name: azure-cluster-template  # Azure-specific

  workers:
    machineDeployments:
    - class: default-worker
      template:
        bootstrap:
          ref:
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
            kind: KubeadmConfigTemplate
            name: worker-bootstrap-template
            namespace: cluster-templates  # Same shared template
        infrastructure:
          ref:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: AzureMachineTemplate
            name: azure-worker-template  # Azure-specific

Managing Cluster Class Lifecycle

Listing Available Cluster Classes

kubectl get clusterclasses -A

Viewing Cluster Class Details

kubectl describe clusterclass vsphere-standard -n capi-templates-vsphere

Updating a Cluster Class

A ClusterClass update affects only new clusters. Existing clusters continue using their original configuration:

kubectl edit clusterclass vsphere-standard -n capi-templates-vsphere

Deleting Clusters Created from Cluster Class

Always delete clusters before removing the ClusterClass:

# Delete the cluster
kubectl delete cluster my-cluster

# Wait for cleanup
kubectl wait --for=delete cluster/my-cluster --timeout=10m

# Then safe to delete ClusterClass if no longer needed
kubectl delete clusterclass vsphere-standard -n capi-templates-vsphere

Template Versioning Strategies

When managing ClusterClasses across environments, consider these versioning approaches:

Semantic Versioning in Names

metadata:
  name: vsphere-standard-v1-2-0
  namespace: capi-templates-vsphere

Using Labels for Version Tracking

metadata:
  name: vsphere-standard
  namespace: capi-templates-vsphere
  labels:
    version: "1.2.0"
    stability: "stable"
    tier: "standard"

Namespace Separation

kubectl create namespace clusterclass-v1
kubectl create namespace clusterclass-v2

This enables gradual migration between ClusterClass versions while maintaining compatibility.

Further Reading

• Cluster API ClusterClass Documentation
• Kamaji Control Plane Provider Reference
• CAPI Provider Integration