Serving LLMs: what actually decides your latency

Training a model is a throughput problem. Serving one is a latency problem under concurrency — and almost every interesting engineering decision in an inference stack falls out of that difference. This is a working note on the pieces that actually decide what your users feel: the phases of a request, the metrics worth measuring, the memory the KV cache quietly eats, what changes when you serve fine-tuned models instead of base ones, and how the popular runtimes — vLLM, SGLang, llama.cpp, MLX — approach the same problem.

TL;DR — Prefill is compute-bound; decode is bandwidth-bound. TTFT is a prefill problem, TPOT is a bandwidth problem, the KV cache is a memory bill that grows with context × concurrency, and multi-LoRA decides whether your GPU count scales with your number of fine-tuned variants. Everything below is those four sentences, unpacked.

1. A request has two phases, and they are not alike

Every LLM request runs in two distinct phases.

Prefill processes the entire prompt — system message, context, user question — in one pass. Because all prompt tokens are available at once, this is a large parallel matrix multiply: compute-bound. Its cost grows roughly linearly with prompt length for the dense work, plus the quadratic attention term over the sequence.

Decode generates the reply one token at a time. Each new token needs a full pass over the model, which means reading every weight from GPU memory to produce a single token. That makes decode memory-bandwidth-bound — the arithmetic units spend most of their time waiting on data.

This split explains most of what follows: optimizations for serving are really optimizations for one phase or the other.

2. The metrics that matter

The four numbers worth putting on a dashboard:

• TTFT — time to first token. Request sent → first streamed token. Dominated by prefill (plus queueing). This is perceived latency: chat UIs feel instant at 300ms TTFT even if the full answer takes twenty seconds.
• TPOT — time per output token (sometimes ITL, inter-token latency). The steady-state rhythm of the stream after the first token. Its inverse is your per-request decode speed.
• Throughput — total tokens/second or requests/second across all concurrent users. What the operator optimizes — in direct tension with per-request latency, because the trick that raises throughput (bigger batches) usually raises TPOT too.
• Goodput — throughput that counts: only requests meeting your latency SLO. Serving at 95% GPU utilization with 30% of requests over TTFT budget can be worse than running cooler with better scheduling.

E2E / TTOT — end-to-end time — is roughly:

Worked example. Say your stack reports TTFT 250ms and TPOT 20ms for a request with a 1,000-token prompt and a 512-token completion:

• TTFT = 250ms
• decode of 511 remaining tokens ≈ 511 × 20ms ≈ 10.2s
• E2E ≈ 10.5s

One number — “tokens per second” — hides all of this. A stack can halve TPOT and leave TTFT untouched; whether that helps depends on whether your users are reading a stream or waiting for a JSON blob.

3. Model size: the memory bill

The weights set the floor. A useful rule: gigabytes ≈ parameters × bytes-per-parameter.

Quantization trades a little quality for a lot of memory — and memory buys speed here, because decode is bandwidth-bound. A back-of-envelope decode ceiling for one stream:

A 7B model at FP16 (14 GB) on an A100 (2 TB/s) tops out near 140 tokens/s per stream — real-world numbers land lower. Quantize to INT4 (3.5 GB) and the same bandwidth now feeds ~4× more streams, or a faster single stream. This is why quantization is a serving decision, not just a “fit it on the GPU” decision.

The quantization formats you’ll actually meet:

• GPTQ / AWQ — 4-bit weight formats built for GPU serving; behind most INT4 checkpoints on the Hub.
• GGUF k-quants — llama.cpp’s format family, tuned for CPU / Metal execution from one file.
• FP8 — hardware-accelerated on Hopper-class NVIDIA chips and newer; near-FP16 quality at half the memory and double the effective bandwidth.
• MLX 4-bit — the Apple-side equivalent, computed on unified memory.

The caveat that matters: quantization quality is eval-dependent. Always re-run your own evals on the quantized checkpoint — a 4-bit model that aces its benchmarks can still lose your production edge cases.

4. The KV cache: the hidden state that eats your GPU

Transformers are autoregressive: each new token attends over everything before it. Recomputing keys and values for the whole history at every step would be quadratic and absurd, so runtimes cache them per sequence. That cache is the KV cache, and after the weights it is the largest consumer of GPU memory — often the largest at real context lengths.

Its size per token:

(the 2 is K and V; modern models use GQA/MQA, so n_kv_heads is much smaller than query heads).

Worked numbers:

• Llama-2-7B: 32 layers × 32 heads × 128 head-dim, FP16 → 2 × 32 × 32 × 128 × 2 ≈ 512 KiB/token. A full 4,096-token context: ~2 GB per sequence.
• Llama-3-70B (GQA, 8 KV heads): 2 × 80 × 8 × 128 × 2 ≈ 320 KiB/token → ~1.25 GB at 4k tokens — per concurrent request.

The key mental model: weights are a fixed cost; the KV cache scales with context × concurrency. Serve ten streams of 8k context on that 7B and the cache alone is nearly triple the weights. This is why “what’s the max context I can serve?” is really “how much memory is left after weights and KV at my target batch size?”

The runtime-side mitigations, briefly:

• PagedAttention (vLLM) — store the cache in fixed-size pages instead of contiguous blocks, eliminating the fragmentation that used to waste 60–80% of KV memory; also the basis of its prefix caching.
• GQA / MQA — fewer KV heads (an architecture choice) shrink the cache 4–8× for a small quality cost.
• Quantized KV cache — 8-bit (or lower) K/V halves the bill again.
• Sliding-window attention — cap how far back a token attends; bounded cache by construction.

5. Context length at inference time

“Supports 128k context” is a model property; serving 128k is an economics problem:

• Prefill compute grows quadratically with sequence length — the first token of a 100k-token prompt costs far more than a thousand 1k prompts.
• KV memory grows linearly, and stays resident for the whole generation.
• Long prompts amplify every queueing effect, because a long prefill blocks the compute everyone else needs.

Positional behavior matters too: models trained at 4k don’t magically attend well at 32k. Techniques like RoPE scaling (linear, NTK-aware, YaRN) stretch positional encodings so the model extrapolates — this is what most “long-context” releases actually ship.

The practical lever is prefix caching: identical prompt prefixes (system prompts, few-shot blocks, retrieved RAG context) produce identical KV, so cache and reuse it instead of recomputing per request. For repeated-prefix workloads this is the single biggest TTFT win available — measured results on Apple Silicon showed cached-context TTFT dropping from 21.7s to 0.78s. SGLang generalizes the idea with RadixAttention: cached prefixes live in a radix tree, so any two requests sharing any prefix branch reuse it automatically.

6. Continuous batching

Classic serving batches a fixed set of requests and waits for the longest generation before admitting new ones — every straggler holds the whole batch hostage. Continuous batching (iteration-level scheduling) admits and retires requests at every decode step: finished sequences free their KV pages immediately and queued prompts slot in. It’s the reason modern stacks report several-fold higher throughput than naive batching, and by now it’s table stakes — every runtime below does it.

7. Serving fine-tuned models: LoRA and the multi-adapter problem

Fine-tuning produces variants; serving them is a separate problem. LoRA freezes the base weights and learns a low-rank delta: $\Delta W = BA$ with rank $r \ll d$, applied to attention (and sometimes MLP) projections. Practical adapters are 0.1–1% the size of the base — a rank-16 adapter on a 7B model is tens of megabytes against the base’s 14 GB.

You can serve those variants two ways:

That second shape is multi-LoRA serving, and doing it well is genuinely hard: requests on different adapters must batch together on the same base weights, adapter matrices must be resident and swapped with near-zero overhead, and the KV cache must be shared/paged across everything. The S-LoRA paper framed the memory problem — a unified pool that pages base weights and adapters together — and modern engines implement versions of it:

• vLLM serves many adapters on one base with hot reload and per-request routing — the most mature option, and my default for this pattern.
• SGLang matches it and, in several published benchmarks, edges ahead at scale, especially when tenants share long system prompts (its radix tree amortizes those across adapters).

The rule of thumb: one or two fine-tunes → merge and serve as separate models. Many per-user or per-tenant variants → one base + multi-LoRA serving, or your GPU bill scales with your customer count.

8. The runtimes, at a high level

All four below are excellent; they optimize different points on the same curve.

vLLM

The production default on NVIDIA hardware. PagedAttention for the KV cache, continuous batching, prefix caching, mature multi-LoRA serving with hot-swappable adapters. If you’re serving a popular open model to many users on GPUs, start here.

SGLang

Built around RadixAttention: cached prefixes in a radix tree shared across requests. That makes it exceptional for agentic and structured workloads — multi-turn tool loops, shared system prompts, RAG pipelines where everything shares a prefix. Also very fast at constrained decoding (guaranteed-JSON outputs), which agent stacks love. Benchmarks in 2026 frequently show it matching or beating vLLM, particularly on prefix-heavy and multi-LoRA-at-scale loads.

llama.cpp

The everything-runs-everywhere runtime: GGUF quantized weights, CPU execution with GPU offload, Metal on macOS. Single-stream latency on modest hardware is remarkable, and it’s the right answer for local tools, edge boxes, and embedded deployments. High-concurrency throughput is not its game.

MLX

Apple’s array framework for Apple Silicon (serve via mlx-lm or vllm-mlx). Unified memory means the RAM/GPU split doesn’t exist: a 64 GB Mac can hold a 70B model that would need two data-center GPUs elsewhere. Recent benchmarks show it competitive natively, with prefix caching delivering dramatic TTFT wins on cached contexts. The right choice for Mac-local serving and Apple-first products.

9. The hardware: the third axis

Runtimes don’t run in the abstract. The same model, same version, same settings lands different numbers on different silicon, because kernel coverage and memory systems differ — so hardware is a first-class serving decision, not a procurement detail.

NVIDIA (CUDA)

The default gravity of the ecosystem: every runtime above targets it first, kernel coverage is deepest, and the memory story (HBM bandwidth, NVLink for multi-GPU) is what most serving math assumes. Two NVIDIA-specific layers sit above the runtimes:

• TensorRT-LLM — NVIDIA’s hand-tuned kernel and engine library; often the fastest raw CUDA path, paid for in a heavier build-and-version matrix.
• Dynamo — NVIDIA’s datacenter-scale serving framework, and the clearest expression of the prefill/decode split from §1: disaggregated serving runs prefill and decode as independently scalable GPU pools, transfers the KV state between them, and routes with KV-cache awareness. It orchestrates TensorRT-LLM, vLLM, and SGLang as backends — prefill and decode simply stop fighting over the same GPUs. This only earns its complexity at serious scale, but it’s where big fleets are going.

Newer chips (Hopper, Blackwell) add FP8 and FP4 tensor-core paths — the hardware half of the quantization story above.

AMD (ROCm)

The credible second source. The MI300X puts 192 GB of HBM3 on one card, which changes the sizing math — a 70B FP16 model fits with room for KV to spare. vLLM and SGLang ship ROCm builds, llama.cpp runs via ROCm/HIP or Vulkan, and PyTorch support is real. The honest caveat: kernel coverage and edge-case polish lag CUDA by quarters, so expect rougher edges at exotic configurations — and validate your model on your card before committing a fleet.

Apple Silicon (MLX)

Covered in §8; the hardware note belongs here too — unified memory removes the RAM/VRAM split entirely, which is why a single Mac can serve models that need multi-GPU rigs elsewhere, at bandwidth numbers well below data-center GPUs.

The practical rule: pick the runtime and the hardware together, not sequentially. “vLLM on the cheapest GPUs,” “SGLang on MI300X,” and “llama.cpp on a Mac mini” are three different products — and Dynamo-style disaggregation only earns its keep at datacenter scale.

10. The pluggable layer: cache extensions, quant backends, speculative decoding

Modern engines are platforms — the kernel is only the base, and a set of loadable optimizations rides on top. Three matter most in practice.

KV-cache extensions — LMCache

LMCache adds a cache layer around the runtime’s KV store: it persists and shares KV state across requests and across instances, tiered from GPU → CPU DRAM → local NVMe. The payoff is TTFT on repeated content — chat history, the same RAG corpus, multi-turn agent state — because the prefix’s KV is loaded, not recomputed, and a second replica can reuse what the first one computed. vLLM integrates it through its KV-connector API, which is also the pattern disaggregated stores (Mooncake-style) build on. If your workload re-reads the same context all day — and most agent stacks do — this is the single highest-leverage extension to try.

Pluggable quantization backends

vLLM’s --quantization flag is a plugin point, not a single feature: fp8, int8 (weight-and-activation), awq, gptq, compressed-tensors, even gguf checkpoints — each backed by different kernels (e.g. Marlin kernels for fast INT4 dequant). Two practical notes:

• Match the format to the silicon — FP8 wants Hopper-or-newer tensor cores; INT4/AWQ is the portable choice; compressed-tensors is the output of the llm-compressor pipeline and the smoothest path if you quantize yourself.
• The backend changes the kernels, not just the weights — the same INT4 checkpoint can serve at noticeably different speeds under different kernel paths, so benchmark the flag, not just the checkpoint.

Speculative decoding

Decode is bandwidth-bound — so amortize it. Speculative decoding drafts k candidate tokens cheaply, then verifies them against the big model in one pass: accept what matches, regenerate from the first mismatch. Same output distribution (verification is exact, so it’s lossless), but up to k tokens per forward pass.

Draft sources, cheapest first:

• Prompt lookup / n-gram — copy spans from the prompt itself; free, and shockingly good for summarization, editing, and code.
• Medusa heads — extra decoding heads on the base model predicting several future tokens at once.
• EAGLE-style drafters — a tiny model drafting in feature space; the current quality/acceptance frontier.
• A small draft model — the classic setup; pick one from the same family.

The catch: the win scales with the acceptance rate. Predictable output (code continuation, structured edits, summarization) accepts often and flies; high-entropy creative text rejects drafts and can even lose a little throughput. vLLM, SGLang, and llama.cpp (draft-model mode) all support it — measure on your traffic before betting on it.

11. A practical checklist

1. Measure TTFT and TPOT separately — one SLO each. A single “latency” number will mislead you.
2. Do the KV math at real concurrency — context × batch × KV-bytes-per-token, on top of weights. This decides your GPU before any tuning does.
3. Enable prefix caching — if your prompts share any structure, it’s the cheapest TTFT win available; reach for LMCache when the sharing crosses requests or instances.
4. Quantize deliberately — weights (GPTQ/AWQ INT4, FP8) for bandwidth, KV (FP8) for long context; measure quality on your evals, not vibes.
5. Try speculative decoding where output is predictable — code, edits, summaries; near-free tokens when the acceptance rate is high.
6. Decide merge-vs-adapter early — it’s an architecture decision, not a deployment detail.
7. Load-test with realistic prompt mixes — long prompts change the bottleneck from decode to prefill and queueing.
8. Choose hardware and runtime as one decision — model × quant × runtime × vendor all constrain each other; check the matrix before you rent.

The unifying idea: serving is memory management under a latency budget. Once you see prefill, decode, and the KV cache as separate line items, every runtime feature — paging, radix trees, adapter pooling — reads as a solution to a specific line of that bill.

Further reading

• vLLM — Efficient Memory Management for LLM Serving with PagedAttention
• NVIDIA Dynamo — disaggregated inference serving framework
• LMCache — KV cache layer for LLM serving
• vLLM — speculative decoding
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters
• SGLang — RadixAttention
• llama.cpp
• MLX — array framework for Apple Silicon
• LLM inference servers compared — TensorFoundry
• vLLM vs SGLang vs TensorRT-LLM vs Ollama (2026) — LeetLLM
• vLLM or llama.cpp? — Red Hat Developer
• Native LLM inference at scale on Apple Silicon (arXiv)

Cite this article

If this note was useful in your own writing, cite it as:

@misc{sharma2026servingllms,
  title        = {Serving LLMs: what actually decides your latency},
  author       = {Sharma, Rahul},
  year         = {2026},
  month        = {August},
  howpublished = {\url{https://rahulsharmavishwakarma.github.io/blog/2026/serving-llms-in-production/}},
  note         = {Online; accessed \today}
}

Or in APA style:

Sharma, R. (2026, August 30). Serving LLMs: what actually decides your latency. Rahul Sharma. https://rahulsharmavishwakarma.github.io/blog/2026/serving-llms-in-production/