LLM inference on Kubernetes needs more than a model server in a container. The service must obtain the right hardware, finish loading before it receives traffic, handle changing demand and remain recoverable when a rollout fails.
Start with one model and a defined workload. Then establish a repeatable deployment record and measure the service under realistic requests. This operating guide explains the decisions and acceptance checks that should surround a first deployment.
Plan LLM inference on Kubernetes around the workload
Define the model, supported context length, expected input and output sizes, request rate and latency requirement. Also decide whether the application needs streaming responses. These choices affect capacity before Kubernetes schedules a single Pod.
Measure a representative request mix. A short classification task and a long document response can stress the same server differently. Therefore, a result measured with short prompts should not become the capacity promise for every request.
Choose a serving engine that supports the model and hardware. The official vLLM Kubernetes guide provides native Deployment and Service examples. Use the example matching your hardware as a starting point, then pin the image and model revision you actually validate.
Establish schedulable GPU capacity
For device-plugin-based GPU scheduling, install compatible vendor drivers and the matching device plugin. Then verify that the nodes advertise the expected resource. Kubernetes’s GPU scheduling documentation explains resources such as nvidia.com/gpu and their allocation rules.
The following fragment requests one NVIDIA GPU through a container resource limit. It is only the resource section of a container specification, not a complete deployable manifest:
resources:
limits:
nvidia.com/gpu: 1
With this mechanism, Kubernetes can use the GPU limit as the request when a separate request is absent. If both are present, they must match. Also specify appropriate CPU and memory resources in the real workload.
A one-GPU allocation does not establish that the model fits. Check weights, runtime overhead and the key-value cache used during generation. For larger models, validate the serving engine’s parallelism settings against the available devices.
Deploy LLM inference on Kubernetes with a clear contract
Keep a deployment record with the image digest, model revision, serving arguments, resource settings, storage and health endpoints. Also include the namespace, access path and rollback target. Another operator should be able to identify the exact candidate from that record.
Make the request path explicit: the application reaches an authorized gateway or private endpoint, which routes to the Kubernetes Service and ready model-server Pods. Apply the required access controls before exposing the service outside its intended network.
Keep model-download credentials in the approved secret mechanism. Also review any setting that executes model-supplied code. A demonstration flag should not silently become an accepted production permission.
Next, validate storage and startup behavior. A local model cache can change cold-start time, while shared storage introduces its own access and performance requirements. Measure the configuration you plan to operate.
Give startup and readiness different jobs
Loading weights and initializing a runtime can take longer than an ordinary web application startup. Configure a startup probe with a budget based on measured cold starts. Otherwise, repeated restarts can prevent a healthy candidate from ever finishing initialization.
Readiness answers whether the Pod should receive Service traffic. Liveness answers whether a running container needs a restart. The Kubernetes probe documentation distinguishes these roles; a failed readiness check does not by itself restart the container.
Also test the transition to ready. A successful health response should correspond to a server that can handle the intended inference request. Then test shutdown so in-flight work follows your defined completion or cancellation policy.
Scale LLM inference on Kubernetes from server metrics
CPU use alone can miss pressure on a GPU-backed model server. Track waiting requests, active work and user-visible latency. Google’s GKE autoscaling guidance explains the tradeoffs between queue- and batch-based signals for inference workloads.
Select a scaling signal against your latency and throughput goals, then load-test it. Queue length counts waiting work, so it may stay low while continuous batching still has room. Also inspect the server’s active workload rather than interpreting one metric in isolation.
Pod scaling and node capacity are separate concerns. An increased replica count cannot serve requests if the new Pods remain pending for lack of suitable GPUs. Include node provisioning and model warm-up in your response-time planning.
Finally, test scale-down. Removing capacity too quickly can interrupt work or cause repeated scaling oscillations. Keep enough stable capacity for the workload your service must absorb.
A practical first-deployment acceptance sequence
Use an isolated test environment and representative synthetic requests. The following is a proposed acceptance sequence, not a claim that a cluster was deployed or benchmarked for this article:
- Confirm that the expected GPU resource is available and the Pod can be scheduled.
- Measure a cold start, then confirm that traffic begins only after readiness.
- Send short and long requests; record completion, latency and errors.
- Increase demand and verify both the scaling decision and usable new capacity.
- Restart a serving Pod and inspect the application’s retry behavior.
- Roll back a failed candidate and verify the previous service again.
Also test access from an unauthorized client. A successful request from inside the cluster does not establish the security of an external endpoint. Keep the acceptance evidence tied to the exact release record.
Monitor the service and its operating cost
Record time to first token, completion latency, token throughput, queue pressure, failures and resource usage where the serving engine exposes them. Then connect those measurements to the application’s accepted outcomes.
Include idle GPU capacity, storage and supporting operations in the cost boundary. Otherwise, a busy load-test result can conceal an expensive lightly used service. Compare full operating periods as well as peak throughput.
Keep a small operational runbook: identify the release, check health and capacity, inspect a failed request, and restore the known working candidate. For practical implementation help, explore Yasa’s development and integration support.
Plan a model service your team can run
Tell Yasa which model you need to serve, the workload it supports and the infrastructure you already use. We can discuss the application integration and a focused next step.
