TL;DR — Prefill is compute-bound; decode is bandwidth-bound. TTFT is a prefill problem, TPOT is a bandwidth problem, the KV cache is a memory bill that grows with context × concurrency, and multi-LoRA decides whether your GPU count scales with your number of fine-tuned variants. Everything below is those four sentences, unpacked.
Every LLM request runs in two distinct phases.
Prefill processes the entire prompt — system message, context, user question — in one pass. Because all prompt tokens are available at once, this is a large parallel matrix multiply: compute-bound. Its cost grows roughly linearly with prompt length for the dense work, plus the quadratic attention term over the sequence.
Decode generates the reply one token at a time. Each new token needs a full pass over the model, which means reading every weight from GPU memory to produce a single token. That makes decode memory-bandwidth-bound — the arithmetic units spend most of their time waiting on data.
| Prefill | Decode | |
|---|---|---|
| What runs | whole prompt, one pass | one token per pass |
| Work shape | large parallel matmuls | full weight read per token |
| Bottleneck | GPU compute | GPU memory bandwidth |
| Cost scales with | prompt length (linear + quadratic attention) | output length × weight size |
This split explains most of what follows: optimizations for serving are really optimizations for one phase or the other.
The four numbers worth putting on a dashboard:
E2E / TTOT — end-to-end time — is roughly:
\[t_{e2e} \approx t_{TTFT} + (n_{out} - 1) \cdot t_{TPOT}\]Worked example. Say your stack reports TTFT 250ms and TPOT 20ms for a request with a 1,000-token prompt and a 512-token completion:
One number — “tokens per second” — hides all of this. A stack can halve TPOT and leave TTFT untouched; whether that helps depends on whether your users are reading a stream or waiting for a JSON blob.
The weights set the floor. A useful rule: gigabytes ≈ parameters × bytes-per-parameter.
| Model | FP16 (2B) | INT8 (1B) | INT4 (0.5B) |
|---|---|---|---|
| 7B | 14 GB | 7 GB | ~3.5 GB |
| 13B | 26 GB | 13 GB | ~6.5 GB |
| 70B | 140 GB | 70 GB | ~35 GB |
Quantization trades a little quality for a lot of memory — and memory buys speed here, because decode is bandwidth-bound. A back-of-envelope decode ceiling for one stream:
\[\text{tokens/s} \;\lesssim\; \frac{\text{memory bandwidth}}{\text{weight bytes}}\]A 7B model at FP16 (14 GB) on an A100 (2 TB/s) tops out near 140 tokens/s per stream — real-world numbers land lower. Quantize to INT4 (3.5 GB) and the same bandwidth now feeds ~4× more streams, or a faster single stream. This is why quantization is a serving decision, not just a “fit it on the GPU” decision.
The quantization formats you’ll actually meet:
The caveat that matters: quantization quality is eval-dependent. Always re-run your own evals on the quantized checkpoint — a 4-bit model that aces its benchmarks can still lose your production edge cases.
Transformers are autoregressive: each new token attends over everything before it. Recomputing keys and values for the whole history at every step would be quadratic and absurd, so runtimes cache them per sequence. That cache is the KV cache, and after the weights it is the largest consumer of GPU memory — often the largest at real context lengths.
Its size per token:
\[\text{KV bytes/token} = 2 \times n_{layers} \times n_{kv\_heads} \times d_{head} \times \text{bytes}\](the 2 is K and V; modern models use GQA/MQA, so n_kv_heads is much smaller than query heads).
Worked numbers:
The key mental model: weights are a fixed cost; the KV cache scales with context × concurrency. Serve ten streams of 8k context on that 7B and the cache alone is nearly triple the weights. This is why “what’s the max context I can serve?” is really “how much memory is left after weights and KV at my target batch size?”
The runtime-side mitigations, briefly:
“Supports 128k context” is a model property; serving 128k is an economics problem:
Positional behavior matters too: models trained at 4k don’t magically attend well at 32k. Techniques like RoPE scaling (linear, NTK-aware, YaRN) stretch positional encodings so the model extrapolates — this is what most “long-context” releases actually ship.
The practical lever is prefix caching: identical prompt prefixes (system prompts, few-shot blocks, retrieved RAG context) produce identical KV, so cache and reuse it instead of recomputing per request. For repeated-prefix workloads this is the single biggest TTFT win available — measured results on Apple Silicon showed cached-context TTFT dropping from 21.7s to 0.78s. SGLang generalizes the idea with RadixAttention: cached prefixes live in a radix tree, so any two requests sharing any prefix branch reuse it automatically.
Classic serving batches a fixed set of requests and waits for the longest generation before admitting new ones — every straggler holds the whole batch hostage. Continuous batching (iteration-level scheduling) admits and retires requests at every decode step: finished sequences free their KV pages immediately and queued prompts slot in. It’s the reason modern stacks report several-fold higher throughput than naive batching, and by now it’s table stakes — every runtime below does it.
Fine-tuning produces variants; serving them is a separate problem. LoRA freezes the base weights and learns a low-rank delta: $\Delta W = BA$ with rank $r \ll d$, applied to attention (and sometimes MLP) projections. Practical adapters are 0.1–1% the size of the base — a rank-16 adapter on a 7B model is tens of megabytes against the base’s 14 GB.
You can serve those variants two ways:
| Approach | How | Memory (50 variants) | Latency | Use when |
|---|---|---|---|---|
| Merge | bake ΔW into the weights | 50 × 14 GB ≈ 700 GB | best — zero overhead | one or two variants |
| Dynamic adapters | one base, attach per request | 14 GB + 50 × ~40 MB ≈ 16 GB | tiny routing overhead | many per-tenant variants |
That second shape is multi-LoRA serving, and doing it well is genuinely hard: requests on different adapters must batch together on the same base weights, adapter matrices must be resident and swapped with near-zero overhead, and the KV cache must be shared/paged across everything. The S-LoRA paper framed the memory problem — a unified pool that pages base weights and adapters together — and modern engines implement versions of it:
The rule of thumb: one or two fine-tunes → merge and serve as separate models. Many per-user or per-tenant variants → one base + multi-LoRA serving, or your GPU bill scales with your customer count.
All four below are excellent; they optimize different points on the same curve.
The production default on NVIDIA hardware. PagedAttention for the KV cache, continuous batching, prefix caching, mature multi-LoRA serving with hot-swappable adapters. If you’re serving a popular open model to many users on GPUs, start here.
Built around RadixAttention: cached prefixes in a radix tree shared across requests. That makes it exceptional for agentic and structured workloads — multi-turn tool loops, shared system prompts, RAG pipelines where everything shares a prefix. Also very fast at constrained decoding (guaranteed-JSON outputs), which agent stacks love. Benchmarks in 2026 frequently show it matching or beating vLLM, particularly on prefix-heavy and multi-LoRA-at-scale loads.
The everything-runs-everywhere runtime: GGUF quantized weights, CPU execution with GPU offload, Metal on macOS. Single-stream latency on modest hardware is remarkable, and it’s the right answer for local tools, edge boxes, and embedded deployments. High-concurrency throughput is not its game.
Apple’s array framework for Apple Silicon (serve via mlx-lm or vllm-mlx). Unified memory means the RAM/GPU split doesn’t exist: a 64 GB Mac can hold a 70B model that would need two data-center GPUs elsewhere. Recent benchmarks show it competitive natively, with prefix caching delivering dramatic TTFT wins on cached contexts. The right choice for Mac-local serving and Apple-first products.
| Pick | When |
|---|---|
| vLLM | Multi-user GPU serving, multi-LoRA, the safe default |
| SGLang | Agentic loops, shared prefixes, structured/JSON output |
| llama.cpp | Local, edge, CPU/mixed hardware, single-stream latency |
| MLX | Apple Silicon, unified-memory serving on Macs |
Runtimes don’t run in the abstract. The same model, same version, same settings lands different numbers on different silicon, because kernel coverage and memory systems differ — so hardware is a first-class serving decision, not a procurement detail.
The default gravity of the ecosystem: every runtime above targets it first, kernel coverage is deepest, and the memory story (HBM bandwidth, NVLink for multi-GPU) is what most serving math assumes. Two NVIDIA-specific layers sit above the runtimes:
Newer chips (Hopper, Blackwell) add FP8 and FP4 tensor-core paths — the hardware half of the quantization story above.
The credible second source. The MI300X puts 192 GB of HBM3 on one card, which changes the sizing math — a 70B FP16 model fits with room for KV to spare. vLLM and SGLang ship ROCm builds, llama.cpp runs via ROCm/HIP or Vulkan, and PyTorch support is real. The honest caveat: kernel coverage and edge-case polish lag CUDA by quarters, so expect rougher edges at exotic configurations — and validate your model on your card before committing a fleet.
Covered in §8; the hardware note belongs here too — unified memory removes the RAM/VRAM split entirely, which is why a single Mac can serve models that need multi-GPU rigs elsewhere, at bandwidth numbers well below data-center GPUs.
| NVIDIA | AMD | Apple | |
|---|---|---|---|
| Ecosystem | CUDA — deepest, default | ROCm — maturing fast | MLX — Apple-only |
| Serve via | TensorRT-LLM, vLLM, SGLang, Dynamo | vLLM / SGLang ROCm builds | mlx-lm, vllm-mlx |
| Memory angle | HBM + NVLink pooling | huge HBM per card (MI300X: 192 GB) | unified memory |
| Quant paths | FP8 / FP4 tensor cores | growing | 4-bit MLX |
The practical rule: pick the runtime and the hardware together, not sequentially. “vLLM on the cheapest GPUs,” “SGLang on MI300X,” and “llama.cpp on a Mac mini” are three different products — and Dynamo-style disaggregation only earns its keep at datacenter scale.
Modern engines are platforms — the kernel is only the base, and a set of loadable optimizations rides on top. Three matter most in practice.
LMCache adds a cache layer around the runtime’s KV store: it persists and shares KV state across requests and across instances, tiered from GPU → CPU DRAM → local NVMe. The payoff is TTFT on repeated content — chat history, the same RAG corpus, multi-turn agent state — because the prefix’s KV is loaded, not recomputed, and a second replica can reuse what the first one computed. vLLM integrates it through its KV-connector API, which is also the pattern disaggregated stores (Mooncake-style) build on. If your workload re-reads the same context all day — and most agent stacks do — this is the single highest-leverage extension to try.
vLLM’s --quantization flag is a plugin point, not a single feature: fp8, int8 (weight-and-activation), awq, gptq, compressed-tensors, even gguf checkpoints — each backed by different kernels (e.g. Marlin kernels for fast INT4 dequant). Two practical notes:
compressed-tensors is the output of the llm-compressor pipeline and the smoothest path if you quantize yourself.Decode is bandwidth-bound — so amortize it. Speculative decoding drafts k candidate tokens cheaply, then verifies them against the big model in one pass: accept what matches, regenerate from the first mismatch. Same output distribution (verification is exact, so it’s lossless), but up to k tokens per forward pass.
Draft sources, cheapest first:
The catch: the win scales with the acceptance rate. Predictable output (code continuation, structured edits, summarization) accepts often and flies; high-entropy creative text rejects drafts and can even lose a little throughput. vLLM, SGLang, and llama.cpp (draft-model mode) all support it — measure on your traffic before betting on it.
The unifying idea: serving is memory management under a latency budget. Once you see prefill, decode, and the KV cache as separate line items, every runtime feature — paging, radix trees, adapter pooling — reads as a solution to a specific line of that bill.
If this note was useful in your own writing, cite it as:
@misc{sharma2026servingllms,
title = {Serving LLMs: what actually decides your latency},
author = {Sharma, Rahul},
year = {2026},
month = {August},
howpublished = {\url{https://rahulsharmavishwakarma.github.io/blog/2026/serving-llms-in-production/}},
note = {Online; accessed \today}
}
Or in APA style:
]]>Sharma, R. (2026, August 30). Serving LLMs: what actually decides your latency. Rahul Sharma. https://rahulsharmavishwakarma.github.io/blog/2026/serving-llms-in-production/
This post is a technical walkthrough of our IJETS paper, Beyond Imagery: AI-Enhanced Diagnostic Assistant for Cancer and Tumor Diagnosis using Radiology Imaging (NandhaGopal S M, Rahul Sharma, Nithin M, Prajwal B R, Prashanth Kalgonda — International Journal On Engineering Technology and Sciences, Vol. XI, Issue III, March 2024). It was my final-year undergraduate research project at HKBK College of Engineering.
A radiologist looking at a CT scan is not just pattern-matching pixels against a list of diseases. She is interrogating the image — Is there a mass in the upper lobe? Is it enhancing? Has it grown since the last scan? — and answering by fusing what she sees with years of accumulated medical knowledge.
Most computer vision systems in medicine do none of that. They classify: this image is “malignant” or “benign”, with a confidence score. Useful, but rigid. The clinician cannot ask a follow-up question, and the system cannot explain itself by pointing at the literature it “learned” from.
Our paper asks a simple question: what if the diagnostic assistant could hold a conversation about the image — and ground its answers in biomedical literature at the same time? The answer we landed on is a multimodal generator (LLaVA-Med) wired to a retrieval layer over biomedical literature. Below is the full architecture, how each part works, what it scores, and where it breaks.
The design philosophy is the same one that made parameter-efficient fine-tuning take off: keep the expensive pretrained machinery frozen, and train only the glue. The vision tower and the LLM already know how to see and speak — the adapter’s whole job is to translate image features into tokens the LLM is willing to attend to.
Medical images are not ImageNet photos: grayscale, high resolution, pathology in fine texture. The encoder is a pretrained vision transformer (CLIP ViT-L/14 in the LLaVA-Med lineage; ResNet-18-style CNNs are the classic baseline). It splits the scan into patches and emits one feature vector per patch:
\[\mathbf{Z}_v = f_v(I) \in \mathbb{R}^{N \times d_v}\]where $N$ is the number of patches and $d_v$ the vision width. The encoder is trained/frozen against medical imaging sets such as VQA-RAD and PathVQA, optimizing cross-entropy — so the features it surfaces are the ones that separate pathologies, not natural-image categories.
The question “Is there pleural effusion, and on which side?” is tokenized and passed through a transformer encoder (RNN/LSTM in older designs). Self-attention resolves what the question is actually about — the organ (pleura), the finding (effusion), the demanded output (a side). This produces the query representation used twice downstream: fused with the image for answering, and as the retrieval query for RAG.
This is the heart of the parameter-efficient design. Rather than co-training a giant joint model, a small two-layer MLP projects vision tokens into the LLM’s embedding space:
\[\mathbf{H}_v = \big(\mathrm{GELU}(\mathbf{Z}_v W_1 + \mathbf{b}_1)\big) W_2 + \mathbf{b}_2\]with the hidden layer deliberately smaller than the input embeddings — an information bottleneck that forces the projection to keep only what matters for answering. The first layer compresses each patch token from the vision dimension $d_v$ down to the bottleneck width; GELU is the nonlinearity that lets the mapping bend around the geometry of the language embedding space rather than staying a flat linear rotation; the second layer expands back out to the LLM’s input dimension $d_l$. After training, an LLM that has never seen an image attends to these projected tokens as if they were ordinary word embeddings — that is the whole trick.
Why an adapter and not fine-tuning the encoders themselves? Three reasons, all practical:
The decoder is a frozen transformer LLM (Vicuna in LLaVA-Med; the paper also discusses LLaMA-2 and GPT-3.5-class models). At each step it predicts:
\[P(y_t \mid y_{<t},\; q,\; \mathbf{H}_v,\; \mathcal{D}_q)\]Note the $\mathcal{D}_q$ — the retrieved passages. That conditioning term is the entire difference between a chatbot that remembers medicine and one that looks it up. Training minimizes cross-entropy against ground-truth answers; because only the adapter receives gradients, the LLM’s clinical language ability (and its failures) stay untouched.
The retrieval pipeline runs in four steps:
1. Index. Biomedical literature and case studies (the PMC-15M corpus that LLaVA-Med itself was curriculum-trained on, plus curated medical databases) are chunked and embedded once, offline.
2. Query. For each incoming question, the question encoder’s representation $e_q$ is matched against document embeddings by cosine similarity:
\[\mathcal{D}_q = \operatorname{TopK}_{d \in \mathcal{C}} \; \frac{e_q \cdot e_d}{\lVert e_q \rVert \, \lVert e_d \rVert}\]3. Assemble. The prompt builder stitches together: image tokens from the adapter, the question, and the top-$k$ retrieved passages — in that order, so attention flows from evidence to question.
4. Generate. The LLM answers conditioned on the passages, which is what makes responses citable: claims in the answer trace back to retrievable sources rather than to compressed weights.
This matters double in oncology. First, for credibility — a generated claim a clinician can verify against its source is a different artifact than one she has to take on faith. Second, for freshness — treatment guidelines move faster than model retraining cycles, and retrieval absorbs that drift without touching a single weight.
The paper evaluates on the three standard biomedical VQA benchmarks:
| Dataset | Images | QA pairs | Character |
|---|---|---|---|
| VQA-RAD | 315 | 3,515 | Clinician-authored; head/chest/abdomen; 11 question categories; ~50% closed-ended |
| SLAKE | 642 | 7,000+ | Physician-annotated, knowledge-enhanced; segmentation masks + bounding boxes; bilingual (English subset used) |
| PathVQA | 4,998 | 32,799 | Pathology; location/shape/color/appearance; open- and closed-ended |
Metrics. Accuracy for closed-set questions; recall (fraction of ground-truth tokens present in the generated answer) for open-set questions.
Base model provenance. LLaVA-Med is curriculum-trained over the PMC-15M figure–caption corpus and fine-tuned on studies from MIMIC-CXR, reaching BLEU-4 of 0.264 and precision of 0.311 on its report-generation objective — the numbers that establish the floor our VQA task builds on.
Three findings from the paper’s comparison against supervised and generative baselines:
Point 3 is the most useful result in the paper. Closed evaluation hides failure modes — with five options, a lucky guess and grounded reasoning score identically. Open generation exposes the gap, and it is precisely the gap retrieval is meant to close: give the decoder something concrete to condition on, and fluency stops being a liability.
The paper’s limitation is its roadmap. Open-set performance is the real test of clinical usefulness, and the fix is not a bigger language model — it is better grounding: stronger retrieval (hybrid dense + keyword, reranking), domain-specific knowledge integration, and evaluation that rewards correct-and-supported answers over merely plausible ones. The first step of that direction is architectural: train only a lightweight projector on a frozen backbone, and let retrieval carry the domain knowledge.
]]>NandhaGopal S M, Rahul Sharma, Nithin M, Prajwal B R, Prashanth Kalgonda, “Beyond Imagery: AI-Enhanced Diagnostic Assistant for Cancer and Tumor Diagnosis using Radiology Imaging,” International Journal On Engineering Technology and Sciences (IJETS), Vol. XI, Issue III, March 2024, pp. 41–46. ISSN (P) 2349-3968, ISSN (O) 2349-3976. Available: PDF ↗