Gateway API

Kamaji provides built-in support for the Gateway API, allowing Tenant Control Planes to be exposed as SNI-based addresses/urls. This eliminates the need for a dedicated LoadBalancer IP per TCP. A single Gateway resource can be used for multiple Tenant Control Planes and provide access to them with hostname-based routing (like https://mycluster.xyz.com:6443).

You can configure Gateway in Tenant Control Plane via tcp.spec.controlPlane.gateway, Kamaji will automatically create a TLSRoute resource with corresponding spec. To make this configuration work, you need to ensure gateway exists (is created by you) and tcp.spec.controlPlane.gateway points to your gateway.

We will cover a few examples below on how this is done.

Prerequisites

Before using Gateway API mode, please ensure:

Gateway API CRDs are installed in your cluster (Required CRDs: GatewayClass, Gateway, TLSRoute)
A Gateway resource exists with appropriate configuration (see examples in this guide):
- Listeners for kube-apiserver.
- Use TLS protocol with Passthrough mode
- Hostname (or Hostname pattern) matching your Tenant Control Plane hostname
(optional) DNS is configured to resolve hostnames (or hostname pattern) to the Gateway's LoadBalancer IP address. (This is needed for worker nodes to join, for testing we will use host entries in /etc/hosts for this guide)
Gateway controller is running (e.g., Envoy Gateway, Istio Gateway, etc.)

To replicate the guide below, please install Envoy Gateway.

Next, create a gateway resource:

Gateway Resource Setup

Your Gateway resource must have listeners configured for the control plane. Here's an example Gateway configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gw-class
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: default
spec:
  gatewayClassName: envoy-gw-class
  listeners:
  - name: kube-apiserver
    port: 6443
    protocol: TLS
    hostname: 'tcp1.cluster.dev'
    tls:
      mode: Passthrough
    allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: TLSRoute
      namespaces:
        from: All

The above gateway is configured with envoy gateway controller. You can achieve the same results with any other gateway controller that supports TLSRoutes and TLS passthrough mode.

The rest of this guide focuses on TCP.

TenantControlPlane Gateway Configuration

Enable Gateway API mode by setting the spec.controlPlane.gateway field in your TenantControlPlane resource:

apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: tcp-1
spec:
  controlPlane:

    # ... gateway configuration:
    gateway:
      hostname: "tcp1.cluster.dev"
      parentRefs:
      - name: gateway
        namespace: default
        sectionName: kube-apiserver
        port: 6443
      additionalMetadata:
        labels:
          environment: production
        annotations:
          example.com/custom: "value"

    # ... rest of the spec
    deployment:
      replicas: 1
    service:
      serviceType: ClusterIP
  dataStore: default
  kubernetes:
    version: v1.29.0
    kubelet:
      cgroupfs: systemd
  networkProfile:
    port: 6443
    certSANs:
    - "tcp1.cluster.dev"  # make sure to set this.
  addons:
    coreDNS: {}
    kubeProxy: {}

Required fields:

hostname: The hostname that will be used for routing (must match Gateway listener hostname pattern)
parentRefs: Array of Gateway references

Optional fields:

additionalMetadata.labels: Custom labels to add to TLSRoute resources
additionalMetadata.annotations: Custom annotations to add to TLSRoute resources

Info

Verify

From our kubectl client machine / remote machines we can access the cluster above with the hostname.

Step 1: Fetch load balancer IP of the gateway.

kubectl get gateway gateway -n default -o jsonpath='{.status.addresses[0].value}'

Step 2: Add host entries in /etc/hosts with the above hostname and gateway LB IP.

echo "<LB_IP> tcp1.cluster.dev" | sudo tee -a /etc/hosts

Step 3: Fetch kubeconfig of tcp cluster.

kubectl get secrets tcp-1-admin-kubeconfig -o jsonpath='{.data.admin\.conf}' | base64 -d > kubeconfig

Step 4: Test connectivity:

kubectl --kubeconfig kubeconfig cluster-info

Multiple Tenant Control Planes

We can use the same Gateway resource for multiple Tenant Control Planes by using different hostnames per tenant cluster. Let us extend the above example for multiple tenant control planes behind single gateway (and LB IP).

# Gateway with wildcard hostname
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
spec:
  listeners:
  - name: kube-apiserver
    port: 6443
    # note: we changed to wildcard hostname pattern matching
    # for cluster.dev
    hostname: '*.cluster.dev'

  # ...
---
# Tenant Control Plane 1
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: tcp-1
spec:
  controlPlane:
    gateway:
      hostname: "tcp1.cluster.dev"
      parentRefs:
      - name: gateway
        namespace: default
  # ...
---
# Tenant Control Plane 2
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: tcp-2
spec:
  controlPlane:
    gateway:
      hostname: "tcp2.cluster.dev"
      parentRefs:
      - name: gateway
        namespace: default
  # ...

Each Tenant Control Plane needs to use a different hostname. For each TCP, Kamaji creates a TLSRoutes resource with the respective hostnames, all TLSRoutes routing through the same Gateway resource.

Konnectivity

If konnectivity addon is enabled, Kamaji creates a separate TLSRoute for it. But this is hardcoded with the listener name konnectivity-server and port 8132. All gateways mentioned in spec.controlPlane.gateway.parentRefs must contain a listener with the same configuration for the given hostname. Below is example configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: default
spec:
  gatewayClassName: envoy-gw-class
  listeners:
  - ...
  - ...
  - name: konnectivity-server
    port: 8132
    protocol: TLS
    hostname: 'tcp1.cluster.dev'
    tls:
      mode: Passthrough
    allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: TLSRoute
      namespaces:
        from: All