LLM Inference on Kubernetes: GPU Scheduling, Autoscaling and Cost

A request has two different accelerator workloads

The client’s support platform defines a complete model replica before assigning cost to capacity.

Prefill processes the prompt, constructs the attention key/value cache, and prepares the first generated token. Many prompt positions can be processed together, so substantial matrix work can expose useful parallelism. Decode then advances each active sequence through successive generation steps, reading prior KV state and appending new state. A short prompt with a long answer and a long prompt with a short answer can consume very different resources even when their total token counts match.

For a conventional autoregressive decoder, each new generation step depends on prior output. Low-batch decode often places pressure on weight and cache memory traffic; large prefills may expose more arithmetic work. These are workload tendencies, not laws that identify the bottleneck from the phase name alone. Batch shape, attention implementation, precision, context length, speculative decoding, and communication all change the balance. Inspect a representative execution trace rather than declaring every prefill compute-bound and every decode bandwidth-bound.

Define latency at the boundary the user experiences

The user does not experience an isolated kernel. You define latency at the boundary where the answer is actually received.

Define client time to first token, or TTFT, as elapsed time from the chosen request-send boundary to receipt of the first output token. It includes transport, frontend processing, admission waiting, prefill, and first-token delivery. Server-side TTFT has a different start point and must be labeled separately. A growing admission queue can ruin client TTFT while an individual prefill kernel remains fast.

For a response with N output tokens and N greater than one, a useful per-request time-per-output-token definition is TPOT = (time of last token - time of first token) / (N - 1), in seconds per output token. Inter-token latency instead describes individual gaps. A per-request average can hide a single long stall, and a streamed chunk may contain multiple tokens; record tokenization and chunk-timestamp conventions. A one-token response has TTFT but no post-first-token TPOT under this definition.

Track client TTFT, per-request TPOT, individual gap distributions, end-to-end completion latency, failures, and queue residence by workload class. The vLLM metrics documentation describes server-side TTFT, inter-token latency, prefill/decode duration, queue duration, waiting requests, and cache occupancy. Metric names and observation boundaries depend on the pinned engine version. Do not add unrelated p99 stage values and call the sum end-to-end p99; retain per-request observations or measure the end-to-end distribution directly.

Useful tokens beat a utilization-only objective

The GPU utilization chart is only supporting evidence. Rejected or late work cannot improve the useful-token objective.

A busy GPU can be recomputing evicted state, serving outputs after their deadlines, or processing work that the client already canceled. Device utilization is a diagnostic signal about sampled activity, not a count of useful answers. High memory allocation may also reflect an engine-reserved cache pool rather than active token occupancy. Separate physical allocation from occupied KV blocks and from requests that actually finish.

For this exercise, define useful output-token throughput as output tokens in successfully completed responses that satisfy the chosen TTFT and TPOT limits, divided by the full observation interval. Also report successful requests per second and application-quality checks: verbosity should not be rewarded as efficiency. Publish all failures, timeouts, cancellations, rejected requests, and requests still pending at the cutoff. None should quietly disappear from the offered-load denominator.

Compare cost per useful output token only within comparable model quality, tokenizer, prompt/output distributions, and service objectives. Include host CPUs, accelerator reservation time, networking, storage, idle warm replicas, and failed work in the cost boundary. A lower allocation is not necessarily a lower bill, and a cheaper token that fails the application task is not an improvement.

Continuous batching is an admission policy, not free capacity

The serving queue becomes an admission decision. Continuous batching must fit both resource capacity and the latency contract.

With continuous batching, the engine can update the active request set between scheduling iterations rather than waiting for the longest sequence in a fixed batch to finish. Completed sequences release their slots and new work can join. This reduces some padding and idle-slot costs, but it does not remove the device-memory budget or make every batch shape equally fast. vLLM documents continuous batching and chunked prefill as distinct serving capabilities.

Prefill competes with active decode work. Large new prompts admitted without a scheduling budget can lengthen token gaps for existing streams. Chunking prefill spreads prompt work across iterations, trading scheduling overhead and possible first-token delay against decode responsiveness. Tune token budgets and sequence limits together with the traffic mix; measure the effect instead of copying a maximum batch size from another model.

Bound both queued work and active work. Define maximum prompt length, maximum generated length, request deadlines, and cancellation behavior before raising concurrency. If admission repeatedly exceeds cache capacity, preemption, recomputation, or offload can turn extra accepted requests into worse goodput. A queue-based scaling signal is useful only if new replicas become ready quickly enough and upstream admission still prevents an unbounded backlog.

Derive the KV budget using KV heads, not query heads

The cache calculation follows KV heads and the declared model shape. A larger query-head count does not automatically mean the same cache layout.

For the stated unsharded full-attention model, KV bytes per cached token = 2 × L × Hkv × D × b. The factor two stores keys and values; L is the number of attention layers; Hkv is the number of key/value heads; D is the per-head dimension; and b is bytes per cached element. Grouped-query attention, or GQA, shares KV heads among more query heads. Use Hkv rather than the query-head count. The formula describes tensor storage, not an allocator footprint.

With L = 32, Hkv = 8, D = 128, and b = 2, the result is 131,072 bytes per token, or 128 KiB. A sequence holding 8,192 total cached tokens uses 1 GiB in this idealized calculation. Here KiB means 2^10 bytes and GiB means 2^30 bytes. Total cached tokens include the prompt and generated history, not just the answer. Holding the other inputs fixed, 32 KV heads would require four times as much cache; this is an architectural comparison, not permission to change a trained model by editing a serving flag.

Subtract weights and the declared reserve before dividing: available KV GiB = max(0, device GiB - weights GiB - reserve GiB). Under the reader defaults, 80 - 16 - 8 = 56 GiB, so the idealized memory-only ceiling is floor(56 / 1) = 56 simultaneous 8,192-token sequences. Eight such sequences consume 8 GiB of KV and 32 GiB including the weights and reserve. Neither number establishes a latency-safe batch size or a throughput result.

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_element
kv_gib_per_sequence = kv_bytes_per_token * cached_tokens / 2^30
available_kv_gib = max(0, device_gib - weights_gib - reserve_gib)
memory_only_sequence_ceiling = floor(available_kv_gib / kv_gib_per_sequence)

Treat the capacity reader as a bound with exclusions

You inspect the capacity reader with its exclusions visible. Its result is a conditional bound, not an allocation promise.

The companion reader deliberately holds layer count and head dimension fixed so the effect of KV-head count, cache precision, context length, and concurrency stays visible. Doubling cached length doubles KV storage in this model. Changing the element size from two bytes to one halves raw tensor storage, but real one-byte cache formats need engine and hardware support, scaling metadata, and quality validation. Weight quantization and KV-cache quantization are separate decisions.

Reserve memory for peak prefill activations, temporary workspaces, captured execution graphs, communication buffers, allocator rounding, and runtime state. Paged-cache blocks can have partially used tails; prefix caching can share some blocks but depends on actual reuse. Sliding-window, hybrid-attention, compressed-cache, and latent-attention models need different accounting. The reader excludes those effects and makes no estimate of fragmentation, prefix hits, or throughput.

Do not divide this result blindly by a tensor-parallel degree. Some implementations shard KV heads while others replicate them for particular head counts or parallel layouts. Inspect the selected model and engine placement. If weights plus reserve exceed device memory, the clamped available-cache value is zero; even an empty KV cache does not make that configuration deployable. A positive cache budget is only one necessary condition for serving.

01 / Follow the explanation

How many sequences fit before the KV cache runs out?

A brief visual sequence plays automatically. The example and its assumptions are already here—nothing to configure.

An interactive teaching model drawn from real delivery work. No agent, cloud account, GPU or cluster is accessed.

Read the full explanation and assumptions ↗

Current example

The requested sequences fit this memory budget

This uniform decoder-only attention model budgets weights, reserve and both key/value tensors at the full selected context. It models memory, not serving throughput or latency. GiB means 2^30 bytes. Tensor parallelism, prefix sharing, sliding windows, paging fragmentation and allocator behavior are omitted. A zero cache budget is an exhausted state; negative headroom shows the shortfall.

Memory-only capacity (sequences)

56 sequences

Example: 80 GiB total, 16 GiB weights, 8 GiB non-KV reserve; 32 layers, 8 KV heads, 128 dimensions/head, 2 bytes/element, 8,192 retained tokens per sequence and 8 sequences. These are model inputs, not a named accelerator or deployed model.

01

Memory envelope

Total device memory (GiB)

80

Weights allocation (GiB)

16

Non-KV reserve (GiB)

8

Available KV budget (GiB)

56.000 GiB

02

Sequence demand

Retained tokens per sequence

8192

Concurrent sequences

8

KV storage per sequence (GiB)

1.000 GiB

Total memory required (GiB)

32.000 GiB

Remaining headroom (GiB)

48.000 GiB

03

Cache representation

KV heads per layer

8 KV heads

KV storage width

2 bytes per element

KV storage per token (bytes)

131072 bytes

GPU resources follow an extended-resource contract

Readiness becomes the bridge between paid devices and accepted assistant responses.

In the traditional device-plugin path covered here, node drivers and the vendor device plugin must already expose a resource such as nvidia.com/gpu. Kubernetes schedules that declared resource; it does not infer free model capacity from an utilization chart. The Kubernetes GPU scheduling documentation permits a GPU limit without an explicit request, in which case the request defaults to the limit. If both are present they must be equal; a GPU request without a limit is not the supported pattern.

These device quantities are integer allocation units, not fractional CPU-style shares. The meaning of one unit still depends on the advertised device-plugin configuration: it may represent exclusive device access, a MIG profile, or shared time-sliced access. Check resource names, discovered accelerator labels, and node-sharing policy. A container memory limit governs host memory, not an independently enforced HBM quota. CPU and host-memory sizing still matter for tokenization, model loading, and the serving frontend.

MIG and time-slicing solve different isolation problems

Sharing the device introduces another decision. MIG and time slicing do not provide the same isolation or memory behavior.

NVIDIA documents MIG as hardware partitioning into predefined GPU instances with memory and fault isolation at that layer. A chosen profile exposes only its portion of memory and compute, and available profiles depend on the device and configuration. Capacity must fit inside the assigned instance. This is not immunity to a host failure, device-wide maintenance, or every possible driver and platform fault.

Time-slicing instead oversubscribes a device and interleaves work from sharing processes. NVIDIA explicitly distinguishes it from MIG: time-sliced replicas do not provide memory or fault isolation. Requesting two advertised replicas does not guarantee twice the compute share, and an integer resource limit is not a proportional-performance contract. A noisy neighbor can alter latency even when each pod appears correctly scheduled.

Use the isolation and predictability required by the workload to choose a deployment policy, then test the allowed colocations under equivalent load. Document shared versus exclusive access in benchmark metadata. Neither packing more pods onto a GPU nor selecting a smaller MIG slice is evidence of savings until useful throughput, tail latency, and recoverability remain acceptable.

A started process is not a model-ready replica

A process starts, but the model still has work to do before serving. Readiness needs to represent that real state.

Startup, readiness, and liveness probes have different jobs. A startup probe delays readiness and liveness checks until startup succeeds, allowing bounded initialization time. Readiness determines whether normal Service routing should send new traffic; liveness addresses failures for which restarting the process is appropriate. A listening socket or a model-name listing does not universally prove that weights, tokenizer, execution graphs, and a usable worker are ready.

The fragment below belongs inside an already defined container; it is not a deployable manifest. The application implements /readyz on port 8000 as a cheap check of engine state that becomes successful only after verified model loading and a bounded warm-up, and /livez as a process-health check independent of queue saturation. These are declared application endpoint contracts, not promises about vLLM endpoints. Verify equivalent semantics for the exact engine version before adapting the fragment.

The example permits roughly 600 seconds of unsuccessful startup checks, polls readiness every five seconds, and uses the host resources from this deployment. Choose the startup allowance from artifact fetch and cold initialization observations, not from warm inference latency. Do not execute an expensive generation on every probe or use transient overload as a liveness failure: restart storms discard warm state and reduce capacity when it is most needed. Readiness also needs an explicit draining state for shutdown; it does not by itself finish existing streams.

# Container fields only; application endpoint contracts described above.
resources:
  requests:
    cpu: "4"
    memory: "24Gi"
  limits:
    memory: "32Gi"
    nvidia.com/gpu: 1
startupProbe:
  httpGet:
    path: /readyz
    port: 8000
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 60
readinessProbe:
  httpGet:
    path: /readyz
    port: 8000
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
livenessProbe:
  httpGet:
    path: /livez
    port: 8000
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Place the parallel group, not just the pod count

Parallel replicas need compatible device groups and communication paths. Counting Pods alone cannot prove the placement.

A model that fits comfortably on one device can be replicated to serve independent requests. Tensor parallelism instead splits operations and introduces communication among devices during execution; pipeline parallelism splits layers and introduces stage transfers and scheduling bubbles. Both may enable a model that otherwise does not fit, but neither guarantees better latency. Replicas and model shards are different capacity units.

Trace the actual communication path: same-device memory, intra-host GPU links, host NUMA placement, NIC attachment, and cross-host fabric. A count of available GPUs does not describe those links. Place a collective-heavy group on the intended topology and ensure all its workers can become ready together. Verify shared-memory and communication-library requirements for the pinned engine without casually widening pod privileges.

Disaggregating prefill and decode can separate their scheduling pressures, but moving KV state becomes a real transfer with bandwidth, lifetime, and failure costs. Record the transfer boundary and its effect on TTFT before assuming phase separation helps. Across any parallel layout, a lost worker may invalidate an entire model group, so recovery headroom must be counted in groups rather than individual pod replicas.

Benchmark the service at controlled offered load

The service meets controlled offered load. You retain queueing, rejected work and latency rather than optimize a utilization headline.

Write the experiment contract first: model and tokenizer digests, engine/runtime/driver versions, device model and sharing mode, parallel layout, weight and KV precision, sampling settings, prompt-length and output-length distributions, prefix reuse, and quality acceptance criteria. Keep cold artifact loading, cold compilation or graph capture, warm steady-state serving, and burst recovery in separate reported phases. Do not put warm-up requests into a steady-state result without saying so.

Use an open-loop arrival schedule to study a specified request rate and its queue buildup; ensure the load generator can sustain that schedule without becoming the bottleneck. A closed-loop fixed-concurrency test is useful too, but it reduces offered traffic when responses slow and can conceal overload. Report which design was used. Sweep offered load through and beyond the intended operating range while holding the workload distribution fixed, and include representative bursts rather than only a smooth average.

For each point, retain the duration, offered and completed counts, errors and rejections, client TTFT/TPOT distributions, useful tokens per second, cache occupancy, queue depth and age, preemptions, and memory peaks. Include the drain policy for requests crossing the measurement cutoff, repeated-run variability, and sample counts supporting tail percentiles. Check a representative output sample for correctness or quality. Only compare candidates at a shared service objective; a higher total-token rate with worse deadline compliance is a different operating point, not a free improvement.

Rehearse failure and make the release reversible in Git

The proposed 6,000-hour monthly reduction has to survive peak demand and loss of a failure domain.

A useful validation plan exercises an unavailable model artifact, a checksum mismatch, initialization that exceeds its allowance, cache exhaustion under long contexts, a lost worker, and a traffic step during replacement. Observe readiness, queue growth, cancellation cleanup, partial-stream behavior, and recovery time. Distinguish host-memory OOM kills from engine-reported device-memory allocation failures. An interrupted stream cannot generally resume by blindly replaying a partially delivered answer; define what the client sees and bound retry amplification.

Version the serving image digest, model and tokenizer artifacts, engine settings, GPU resource/profile, parallel topology, probe contract, and routing change as one reviewed release. Start with a controlled candidate allocation and explicit stop conditions based on errors, TTFT/TPOT compliance, queue age, and readiness. Allow old and new replicas to coexist with enough device capacity for the replacement strategy; a surge replica that remains pending cannot provide rollout safety.

A reversible GitOps rollout restores the previous compatible desired state, not merely a mutable model alias. Retain its artifacts and device capacity, and keep tokenizer/API changes compatible while both versions serve. Argo CD documents that direct application rollback is unavailable with automated sync enabled; make a reviewed Git revision that restores the known-good configuration, or follow the controller-specific approved recovery procedure. An imperative live patch may be overwritten by reconciliation. Plan draining and in-flight request outcomes separately from configuration rollback, which cannot undo already delivered tokens. These are proposed exercises only; no release or failure injection is executed here.