Modelplane Modelplane docs
Version
main v0.1 v0.2 v0.3 Latest

ServingStack Custom Resource

On this page

A ServingStack installs the serving substrate on a Kubernetes cluster: LeaderWorkerSet (Standard) or Grove and the KAI Scheduler (Dynamo), plus Gateway API, cert-manager, and Prometheus.

#Metadata

API version
infrastructure.modelplane.ai/v1alpha1
Kind
ServingStack
Scope
Namespaced
Short names
ss

#Example

Manifest
apiVersion: infrastructure.modelplane.ai/v1alpha1
kind: ServingStack
metadata:
  name: west-gke-stack
  namespace: platform
spec:
  secrets:
    - type: Kubeconfig
      name: west-gke-kubeconfig
      key: kubeconfig
    - type: GoogleApplicationCredentials
      name: west-gke-sa-key
      key: private_key
  versions:
    certManager: "v1.21.1"
    envoyGateway: "v1.8.4"
  gateway:
    listeners:
      - name: http
        port: 80
        protocol: HTTP

#Spec

ServingStackSpec defines the desired state of ServingStack.

# dynamo optional object

Version pins for Dynamo-stack components. Ignored when stack is Standard.

# grove optional string 1–32 chars default: v0.1.0-alpha.12-rc2

Grove chart version. Grove gang-schedules a multi-node engine as a PodCliqueSet when stack is Dynamo. Pinned at v0.1.0-alpha.12-rc2 or later: earlier releases append Grove’s own env vars after template env instead of prepending them (grove#753), which blocks Modelplane from aliasing MODELPLANE_LEADER_ADDRESS to them.

# kaiScheduler optional string 1–32 chars default: v0.16.8

KAI Scheduler chart version. Grove hands a PodCliqueSet’s gang-scheduling to KAI, which binds each gang all-or-nothing against a Queue.

# modelExpress optional string 1–32 chars default: 0.4.1

ModelExpress server image tag. The metadata-only server coordinates peer-to-peer weight transfer between engine pods that opt into –load-format modelexpress.

# gateway optional object

Configuration for the cluster’s inference traffic gateway.

# className optional string 1–63 chars default: envoy

GatewayClass name. Override if the cluster already has a GatewayClass named envoy.

# clientCAs optional object[] ≤ 32 items
# certificate required string 1–16384 chars

The CA certificate, PEM encoded.

# name required string 1–253 chars

The InferenceGateway this CA belongs to.

# hostname optional string 1–253 chars

The name this cluster’s gateway is reached by, projected from the InferenceCluster. The gateway serves a certificate for it, so an InferenceGateway can originate TLS and know it reached the right cluster. Without it the gateway serves plain HTTP and carries no traffic, since an InferenceGateway addresses a cluster by name.

# listeners optional object[] ≤ 8 items
# name required string 1–63 chars

Unique listener name.

# port required integer 1–65535

Port number for this listener.

# protocol required enum: HTTP | TCP

Protocol for this listener.

# nvidiaDriverRoot optional string ≤ 512 chars default: /

Host path where the NVIDIA driver is installed, passed to the DRA driver as nvidiaDriverRoot. Defaults to / (the upstream default), which suits EKS and self-managed clusters. Set it for platforms that install the driver elsewhere — GKE uses /home/kubernetes/bin/nvidia. A non-default value also makes the serving stack compose a ResourceQuota permitting the DRA driver’s system-critical pods, which GKE requires. The cluster composition sets this; the serving stack never inspects its own cloud.

# secrets required object[] 1–8 items
# key required string ≤ 253 chars

Key within the Secret that holds the credential data.

# name required string ≤ 253 chars

Name of the Secret.

# namespace optional string ≤ 253 chars

Namespace of the Secret, when it isn’t this ServingStack’s namespace. Set on cloud identity entries whose credential is the Secret the cloud provider’s ProviderConfig references.

# type required enum: Kubeconfig | GoogleApplicationCredentials | AWSWebIdentityCredentials | NebiusServiceAccountCredentials

The type of credential this secret contains. Kubeconfig is required. Any other value is a cloud identity type; when present, the serving stack authenticates to the cluster as that identity instead of using the kubeconfig’s embedded credentials.

# stack optional enum: Standard | Dynamo default: Standard

Which serving stack this installs. Standard (the default) is the Modelplane-composed serving layer: a Deployment or LeaderWorkerSet, Gateway API, and the endpoint picker. Dynamo swaps in NVIDIA’s components: Grove with the KAI Scheduler for multi-node gang scheduling, and a shared ModelExpress server for weight distribution. Propagated from the InferenceCluster.

# standard optional object

Version pins for Standard-stack components. Ignored when stack is Dynamo.

# leaderWorkerSet optional string 1–32 chars default: v0.8.0

LeaderWorkerSet chart version.

# versions optional object

Version pins for each component. Defaults are the latest tested combination. Override individual versions to upgrade components independently.

# certManager optional string 1–32 chars default: v1.21.1

cert-manager chart version.

# envoyGateway optional string 1–32 chars default: v1.8.4

Envoy Gateway chart version. Must support InferencePool backend resources (the disaggregated-serving routing path), which requires v1.8.x or newer; older releases lack the Gateway API CRDs (ListenerSet) the AI Gateway needs. Envoy AI Gateway v1.1.x is tested against Envoy Gateway v1.8.x with Gateway API v1.5.x, so v1.9.x is out of range until the AI Gateway release that pairs with it.

# nodeFeatureDiscovery optional string 1–32 chars default: 0.18.3

Node Feature Discovery chart version. NFD labels GPU nodes so the NVIDIA DRA driver targets its kubelet plugin to them.

# nvidiaDraDriver optional string 1–32 chars default: 0.4.0

NVIDIA DRA driver chart version. Publishes GPUs as DRA ResourceSlices and the gpu.nvidia.com DeviceClass that ModelReplica ResourceClaims bind through.

# prometheus optional string 1–32 chars default: 72.6.2

kube-prometheus-stack chart version.

# trustManager optional string 1–32 chars default: v0.24.0

trust-manager chart version. trust-manager distributes the cluster gateway’s CA certificate without its private key, which is what lets the control plane read the certificate to hand to a fleet gateway.

#Status

# gateway optional object

Status of the cluster’s inference gateway.

# address optional string ≤ 256 chars

The gateway’s external address, once assigned by the cloud load balancer.

# caCertificate optional string ≤ 16384 chars

PEM certificate of the CA that signed this gateway’s serving certificate. An InferenceGateway validates the gateway against it, so it reaches the cluster it meant to and not whatever answers on that address.