Version

Route to External Providers

On this page

API: modelplane.ai/v1alpha1 · ModelEndpoint

A ModelEndpoint is a single reachable inference endpoint that a ModelService can route to. Modelplane creates one for each of your replicas automatically, but you can also create one by hand to point at an inference endpoint Modelplane doesn’t run, most often a SaaS provider like Together or Baseten. A service treats both the same, so you can front your own replicas and an external provider behind one URL: send overflow to the provider when your fleet is busy, or fail over to it as a break-glass option.

Routing to an external provider

Create a ModelEndpoint with three things:

# Modelplane composes a ModelEndpoint per ModelReplica. Write one by hand only
# to register a model it doesn't run, like this one at Together, so a
# ModelService can fan over both.
apiVersion: modelplane.ai/v1alpha1
kind: ModelEndpoint
metadata:
  name: kimi-k2-together
  namespace: ml-team
  labels:
    # 1. A label of your own for a ModelService to select on. Any label works.
    modelplane.ai/endpoint: kimi-k2-together
spec:
  # 2. Scheme and host, no path. An https origin gets TLS originated to it.
  #    This must be a name rather than an address: Envoy AI Gateway only applies
  #    per-backend model rewriting, credentials and priority failover when every
  #    backend in a route is addressed by hostname.
  origin: https://api.together.xyz
  api:
    # 3. The API this backend speaks, and the path it serves it under. The
    #    gateway translates between this and whatever the caller sent, so an
    #    Anthropic client can reach an OpenAI backend and the reverse. Most
    #    providers serve /v1; Groq serves /openai/v1.
    schema: OpenAI
    prefix: /v1
  # 4. The name Together knows this model by. The gateway rewrites the request
  #    body's model to it, so a caller keeps naming the ModelService and gets
  #    back whichever model actually served.
  model: moonshotai/Kimi-K2-Instruct
  # 5. Together's API key, which the gateway attaches on the way out. It never
  #    reaches the caller, and the caller's own key never reaches Together. Nor
  #    does the caller's identity: Modelplane strips that header for any backend
  #    it doesn't operate, while still recording the caller in the usage record.
  credentialRef:
    name: together-api-key

Then point a ModelService at it. Selecting modelplane.ai/external-provider: together routes to the provider; adding a second entry for a deployment fronts both behind one URL, so traffic can spill over to the provider alongside your own replicas:

# A ModelService's endpoints list combines: one entry can select your own
# deployment's replicas while another selects an external ModelEndpoint, so
# overflow or break-glass traffic to a SaaS provider sits behind the same URL
# as your own replicas.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: kimi-k2
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: kimi-k2          # your own replicas
  - selector:
      matchLabels:
        modelplane.ai/external-provider: together  # the endpoint above

Anything speaking the OpenAI or Anthropic API works. origin is the scheme and host to reach it at, with no path; api.prefix is the path the provider serves those APIs under, and api.schema which of the two it speaks. Only those change between providers.