Modelplane Modelplane docs
Version
main v0.1 v0.2 v0.3 Latest

ModelEndpoint Custom Resource

On this page

A ModelEndpoint is somewhere a request can be served: one replica of a ModelDeployment, or a model at a provider like Together or Groq. It describes a backend well enough for a gateway to talk to it without knowing where it came from, which is what lets a ModelService fan over endpoints Modelplane runs and endpoints it merely buys from. Modelplane composes one per replica. You write them by hand for anything it doesn’t run.

Concept guide: Route to External Providers →

#Metadata

API version
modelplane.ai/v1alpha1
Kind
ModelEndpoint
Scope
Namespaced
Short names
me

#Example

Manifest
# ModelDeployment composes a ModelEndpoint per replica. Write one by hand only
# to register a model Modelplane doesn't run, like this one at Together.
apiVersion: modelplane.ai/v1alpha1
kind: ModelEndpoint
metadata:
  name: together-qwen-72b
  namespace: ml-team
  labels:
    modelplane.ai/endpoint: together-qwen-72b
spec:
  # Scheme and host, no path. An https origin gets TLS originated to it. The
  # host must be a name; an address stops the gateway applying the model
  # rewrite, the credential and priority failover.
  origin: https://api.together.xyz
  api:
    # OpenAI (the default) or Anthropic. The gateway translates between this
    # and whatever the caller sent.
    schema: OpenAI
    # The path this backend serves that API under. /v1 for most, /openai/v1 for
    # Groq, a per-replica path for a Modelplane-composed endpoint.
    prefix: /v1
  # The name this backend knows the model by. Unset, the caller's model name
  # passes through unchanged.
  model: Qwen/Qwen2.5-72B-Instruct-Turbo
  # This backend's credential, attached by the gateway on the way out. It never
  # reaches the caller, and the caller's own credential never reaches here.
  credentialRef:
    name: together-api-key

#Spec

# api optional object

The API this backend speaks, and where it serves it. Defaults to the OpenAI API under /v1, which is what most providers and every Modelplane-composed endpoint serve.

# prefix optional string 1–512 chars default: /v1

The path the backend serves that API under: /v1 for most, /openai/v1 for Groq, and a per-replica path for a Modelplane-composed endpoint, whose cluster gateway distinguishes replicas by path.

# schema optional enum: OpenAI | Anthropic default: OpenAI

The API the backend speaks. A gateway translates between this and whatever the caller sent, so an OpenAI client can reach an Anthropic backend and the reverse.

# credentialRef optional object

Secret holding this backend’s credential, which the gateway attaches on the way out. The credential never reaches the caller, and the caller’s own credential never reaches the backend. An endpoint whose Secret is missing carries no traffic and says so in its conditions.

# key optional string 1–253 chars default: apiKey

The Secret key holding the credential.

# name required string 1–253 chars

Secret in this ModelEndpoint’s namespace, with the credential under the key named by key.

# model optional string 1–253 chars

The name this backend knows the model by, which a gateway rewrites the request’s model to on the way out. Unset, the caller’s model name passes through unchanged. A caller names a ModelService and gets back whichever model actually served, the way asking OpenAI for gpt-4o returns gpt-4o-2024-08-06.

# origin required string 1–2048 chars

Scheme and host of the backend, with no path: an https origin gets TLS originated to it. A port is only needed for a non-default one. The host must be a name, never an address. Envoy AI Gateway applies per-backend model rewriting, credentials and priority failover only when every backend in a route is addressed by hostname; given an address it keeps passing traffic but silently stops applying them, which would send a caller’s own model name to a provider with no credential attached.

#Status

# conditions optional object[]