InferenceGateway Custom Resource
An InferenceGateway is the front door for inference requests: the only address a caller sees. It speaks the OpenAI and Anthropic APIs, authenticates callers, resolves the model a request names to a ModelService, and forwards to whichever of that service’s endpoints should serve it, translating the request for the backend that won. A Modelplane can run several, each on an InferenceCluster of its own. Run one per region to keep a caller’s traffic in its jurisdiction, or two in a region to survive losing a cluster. Modelplane runs no global load balancer: distributing callers across gateways is yours to configure, whether by geo DNS, an anycast address, or an edge of your own with these gateways as origins.
Concept guide: Set Up the Gateway →
#Metadata
#Example
Manifest
apiVersion: modelplane.ai/v1alpha1
kind: InferenceGateway
metadata:
name: eu
spec:
# The InferenceCluster this gateway runs on, which decides its region and its
# address. The cluster needs no GPU pools.
clusterName: gw-gcp-eu
# The name the gateway answers on. Point it at status.address.
hostname: eu.example.com
tls:
certificateRefs:
- name: eu-example-com-tls
auth:
# Each key in a selected Secret is one caller: the entry's name is the
# caller's identity, its value is the key. Adding a caller means writing a
# Secret, not editing this gateway.
secretSelector:
matchLabels:
modelplane.ai/inference-keys: "true"
# The ModelServices this gateway serves. Absent, it serves every one. Scoped
# here to a region, which is how residency is expressed.
serviceSelector:
matchLabels:
example.org/region: eu
#Spec
Authenticates callers against keys this gateway holds. Omit it and the gateway authenticates nobody, so anything that can reach the address can invoke any ModelService it serves. That is only appropriate behind something that has already established who is calling. Modelplane authenticates callers; it does not authorize them. Every accepted key can reach every ModelService this gateway serves, and /v1/models lists them all regardless of key. To narrow what a key can reach, narrow the gateway with serviceSelector or run a separate gateway with its own keys.
Selects Secrets holding caller API keys. Each key in a selected Secret is one caller: the entry’s name is the caller’s identity and its value is the key. So adding a caller means writing a Secret, not editing this gateway. The gateway stamps the resolved identity onto every request and every usage record, and never forwards the caller’s key. Ranking one caller above another is not Modelplane’s decision to make, so it publishes the identity and leaves acting on it to whatever does decide.
The InferenceCluster this gateway runs on, which decides its region and its address. A gateway doesn’t move: unlike a ModelDeployment, whose replicas re-place when their cluster goes away, a gateway stays where it was put. Availability comes from running more of them, because failing over would change the address callers use and could move traffic out of the jurisdiction the gateway exists to hold. The cluster needs no GPU pools. A cluster with none is a gateway and nothing else, which is what a region with callers but no accelerators wants. A cluster that serves models can host a gateway too, and does so at most once.
The name this gateway answers on. Point it at status.address once the gateway has one. Omit it and the gateway answers on its address alone, over plain HTTP. That is the getting-started shape, and also the shape for anyone terminating TLS on an edge of their own in front of the gateway.
Selects the ModelServices this gateway serves, by their labels. Absent, it serves every one. This is how a gateway is scoped: to a region, so an EU service is only reachable through EU gateways; to your public services on an internet-facing front door; or to a named set on a dedicated gateway. These are your labels, under your own prefix. Modelplane matches them and never interprets them, so a region means no more to it than any other label.
#Status
The address this gateway answers on, and what spec.hostname should point at. It is also the target to health check, at /healthz, to decide whether this gateway is in rotation. /healthz answers 200 whenever this gateway’s proxy is running and serving. It says nothing about whether any ModelService is reachable through it, so a gateway with no healthy backend stays in rotation and answers requests with a 503. Read each ModelService’s RoutingReady for that.
PEM certificate of the CA that signs this gateway’s client certificate. Every InferenceCluster accepts client certificates from it, which is how this gateway proves itself to a cluster gateway and how anything else is refused. One CA per gateway rather than one per Modelplane, so that no private key has to be distributed: each is generated on the cluster that uses it and only its certificate travels. Note that every cluster gateway trusts every fleet gateway’s CA and checks the signing CA rather than the subject, so this bounds where the keys live, not what one of them can reach.