Kamaji and `externalClusterReference` usage

This document explains how to use Kamaji's externalClusterReference together with Cluster API (CAPI) to run Kubernetes control planes on an external cluster, while managing worker nodes from a management cluster.

It assumes the use of the KubeVirt infrastructure provider for ease of deployment and local testing.

High-level Architecture

The following setup operates on two Kubernetes clusters:

Management cluster – runs Cluster API controllers and the Kamaji control-plane provider, and manages cluster lifecycle and topology.
External cluster - runs Kamaji, hosts the Kubernetes control plane components and receives control plane workloads via Kamaji

Prerequisites

docker
kind
kubectl
clusterctl
helm

Step 1: Create the KIND clusters

Create the management cluster:

kind create cluster --name management

Create the external cluster that will host control planes:

kind create cluster --name external

Verify contexts:

kubectl config get-contexts

Step 2: Initialize Cluster API controllers

Switch to the management cluster:

kubectl config use-context kind-management

Enable ClusterClass support and initialize Cluster API with Kamaji and KubeVirt:

export CLUSTER_TOPOLOGY=true
clusterctl init \
  --core cluster-api \
  --bootstrap kubeadm \
  --infrastructure kubevirt \
  --control-plane kamaji

Step 3: Enable Kamaji external cluster feature gates

Patch the Kamaji controller to enable externalClusterReference:

kubectl -n kamaji-system patch deployment capi-kamaji-controller-manager \
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/0/args/1",
      "value": "--feature-gates=ExternalClusterReference=true,ExternalClusterReferenceCrossNamespace=true"
    }
  ]'

Step 4: Install KubeVirt

Fetch the latest stable KubeVirt version and install:

export VERSION=$(curl -s "https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt")
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml"
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml"

Enable emulation (optional, if virtualization is not supported):

kubectl -n kubevirt patch kubevirt kubevirt \
  --type=merge \
  --patch '{"spec":{"configuration":{"developerConfiguration":{"useEmulation":true}}}}'

Step 5: Prepare kubeconfig for the external cluster

Retrieve the external cluster control-plane address:

EXT_CP_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "external-control-plane")

Export and rewrite the kubeconfig:

kubectl --context kind-external config view --raw --minify --flatten > kind-external.kubeconfig

Replace the API endpoint with the cluster IP, required for cross-cluster access from the management cluster:

bash -c "sed -i -E 's#https://[^:]+:[0-9]+#https://$EXT_CP_IP:6443#g' kind-external.kubeconfig"

Create the kubeconfig secret in the management cluster:

kubectl -n default create secret generic kind-external-kubeconfig \
  --from-file=kubeconfig=kind-external.kubeconfig

Step 6: Install Kamaji and dependencies on the external cluster

Switch context:

kubectl config use-context kind-external

Install cert-manager:

helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

Install Kamaji:

helm upgrade --install kamaji clastix/kamaji \
  --namespace kamaji-system \
  --create-namespace \
  --set 'resources=null' \
  --version 0.0.0+latest

Install MetalLB:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.3/config/manifests/metallb-native.yaml

Configure MetalLB IP address pool:

SUBNET=$(docker network inspect kind | jq -r '.[0].IPAM.Config[] | select(.Subnet | test(":") | not) | .Subnet' | head -n1)
NET_PREFIX=$(echo "$SUBNET" | cut -d/ -f1 | awk -F. '{print $1"."$2}')
kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - ${NET_PREFIX}.255.200-${NET_PREFIX}.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
EOF

Create tenant namespace:

kubectl create namespace kamaji-tenants

Step 7: Definition of KamajiControlPlaneTemplate

The KamajiControlPlaneTemplate is defined in the following manifest and can be applied directly.

This template configures how Kamaji deploys and manages the tenant control plane on an external Kubernetes cluster using Cluster API.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlaneTemplate
metadata:
  name: kamaji-controlplane-external
  namespace: default
spec:
  template:
    spec:
      addons:
        coreDNS: {}
        konnectivity: {}
        kubeProxy: {}
      dataStoreName: "default" # reference to DataStore present on external cluster
      deployment:
        externalClusterReference:
          deploymentNamespace: kamaji-tenants
          kubeconfigSecretName: kind-external-kubeconfig
          kubeconfigSecretKey: kubeconfig
      network:
        serviceType: LoadBalancer
      kubelet:
        cgroupfs: systemd
        preferredAddressTypes:
        - InternalIP
      registry: "registry.k8s.io"

The .spec.template.spec.deployment.externalClusterReference section defines how Kamaji connects to and deploys control plane components into the external cluster:

deploymentNamespace - The namespace on the external cluster where TenantControlPlane resources and control plane components are created.
kubeconfigSecretName - The name of the Kubernetes Secret containing a kubeconfig that allows Kamaji to authenticate to the external cluster.
kubeconfigSecretKey - The key inside the secret that holds the kubeconfig data.

The referenced secret must exist in the Kamaji management cluster and provide sufficient permissions to create and manage resources in the target external cluster.

Step 8: Create the Cluster

Switch context back to the management cluster:

kubectl config use-context kind-management

Apply the Cluster manifest:

kubectl apply -f "https://raw.githubusercontent.com/clastix/kamaji/master/config/capi/clusterclass-kubevirt-kamaji-external.yaml"

Validation

Check tenant control plane pods running in the external cluster:

kubectl --context kind-external -n kamaji-tenants get pods

Check cluster status in the management cluster:

kubectl --context kind-management get clusters
kubectl --context kind-management get kamajicontrolplanes

Get cluster kubeconfig and confirm it is working:

kubectl config use-context kind-management
clusterctl get kubeconfig demo-external > demo-external.kubeconfig
KUBECONFIG=./demo-external.kubeconfig kubectl get nodes

Clean up

Delete Kind clusters:

kind delete cluster --name management
kind delete cluster --name external

Summary

Using externalClusterReference with Kamaji and Cluster API enables:

Hosted Kubernetes control planes on remote clusters
Strong separation of concerns
Multi-cluster management patterns
Clean integration with ClusterClass

Kamaji and externalClusterReference usage