vLLM vs SGLang vs TensorRT-LLM 2026: Inference Engine Comparison

If you are choosing between vLLM vs SGLang and TensorRT-LLM in 2026, the honest one-liner is this: vLLM is the safe default for breadth of models and hardware, SGLang wins on prefix-heavy and agentic workloads and on day-0 frontier-model support, and TensorRT-LLM is the maximum-performance option if you are NVIDIA-only and serving a small number of models for a long time. All three are Apache-2.0, all three now ship the same baseline feature set, and the feature-gap framing from 2023 is dead. What is left is workload shape, hardware, and governance.

This is a serving-layer decision that sits underneath the platform questions we cover in BentoML vs KServe, and one layer above the build-versus-buy call in RL training infrastructure.

The short answer

• Pick vLLM if you serve many models, run mixed or non-NVIDIA hardware, or want the largest community and the shortest path from checkpoint to endpoint.
• Pick SGLang if your traffic is prefix-heavy - multi-turn chat, RAG behind a fixed system prompt, agent trees, self-consistency sampling - or if you need frontier models supported the week they drop.
• Pick TensorRT-LLM if you are NVIDIA-only, serving a stable set of high-value models, and want the deepest co-design with Blackwell-class silicon and rack-scale deployments.

Deciding factor to pick

What each engine is in 2026

vLLM is the most widely deployed open-source LLM inference engine, at roughly 91,000 GitHub stars and 270 contributors in a single release cycle as of September 2026, with v0.28.0 shipping in late August. It has been a PyTorch Foundation hosted project since May 2025, which gives it genuinely neutral governance. One correction worth making because most articles still get it wrong: there is no V0 versus V1 choice any more. Release v0.11.0 in October 2025 removed the V0 engine entirely, and V1 is the only engine in the codebase. The current architectural push is a rewrite of the model execution layer that the Q3 2026 roadmap targets across the top model architectures.

SGLang sits at roughly 35,500 stars with v0.5.19 released in early September 2026. Its two signature ideas are RadixAttention, which stores the KV cache in a radix tree so shared prefixes are matched and reused across requests, and a zero-overhead CPU scheduler that overlaps Python-side scheduling work with GPU compute so the accelerator rarely idles. The unified radix tree became the default in v0.5.19. The governance change is the big 2026 story: SGLang spun out into a company, RadixArk, which launched with a $100 million seed led by Accel at a reported valuation around $400 million post-money, with NVIDIA’s and AMD’s venture arms both participating. It remains Apache-2.0, but “LMSYS non-profit project” is no longer an accurate description.

TensorRT-LLM is NVIDIA’s engine, at roughly 14,500 stars, with v1.2.1 the current stable release as of September 2026 and a long v1.3 release-candidate train behind it. The pivot that matters is version 1.0 in September 2025, which removed the TensorRT backend and made PyTorch the sole execution backend. The old mental model of a mandatory ahead-of-time engine compilation step is obsolete. It installs with pip, serves through an OpenAI-compatible endpoint, and is operationally much closer to its rivals than its reputation suggests.

Head-to-head

Throughput and latency: what the published benchmarks actually say

Here is the part most comparison posts fabricate, so read this section carefully.

There is no neutral, current, three-way head-to-head benchmark of all three engines on identical hardware at identical software dates. None exists. What does exist is worth using properly.

MLPerf Inference v6.0, published 1 April 2026, is the one audited data point. On the gpt-oss-120b benchmark with 8 NVIDIA B200 accelerators in the closed division:

That is roughly 8 percent to vLLM on Offline throughput and 22 percent to TensorRT-LLM on latency-bounded Server. The reading that survives scrutiny: vLLM is now genuinely competitive with NVIDIA’s own engine on NVIDIA’s newest silicon when you are maximising throughput, while TensorRT-LLM keeps an advantage under Server’s latency-constrained arrival process. Caveats you should carry with the numbers: different submitters means different host CPU, memory, and tuning, and the closed division permits different quantization as long as the accuracy target is met.

SGLang did not submit to MLPerf v6.0. Zero results rows. That is a real and checkable asymmetry, and it means its production claims - the project cites over 400,000 GPUs and names xAI, Cursor, LinkedIn and the major clouds among users - rest on project statements rather than audited benchmarks.

The best continuous three-way source is SemiAnalysis InferenceX, an Apache-2.0 open-source harness plus public dashboard that re-runs vLLM, SGLang and TensorRT-LLM across GB300, GB200, B300, B200, H200, H100, MI355X, MI325X and MI300X. Two things to know before you lean on it. First, it is compute-sponsored: the project publicly thanks both AMD and NVIDIA for hardware access, so it is auditable but not compute-independent. Second, it assigns models to engines rather than running everything on everything, and its policy explicitly treats upstream vLLM and SGLang as first-class engines with TensorRT-LLM as a secondary submission, on the stated grounds that hardware-specific engines do not always support every feature the workloads need. That policy is itself the clearest signal of where the open-source center of gravity moved since 2024.

The single most useful line in that project’s documentation is that benchmarks taken at a fixed point in time go stale quickly. Lambda quantified exactly that: a 9 percent throughput improvement on identical hardware between MLPerf v5.1 and v6.0, attributed purely to six months of stack maturity. NVIDIA reports a 2.7x gain on DeepSeek-R1 on GB300 NVL72 across a similar window, attributed to TensorRT-LLM software alone. Any number you read, including the ones above, decays.

One further third-party test is worth citing carefully. AIMultiple ran a disclosed-methodology benchmark on a single H100 with Llama-3.1-8B-Instruct in bfloat16, 1,000 ShareGPT prompts across ten passes, and reported SGLang at 16,215 tokens/second against vLLM with FlashInfer at 12,553. That is a legitimate result for that model, that hardware and that prompt distribution. It is also not a three-way test, because TensorRT-LLM was not included. If you find a blog presenting those exact numbers as a three-way comparison, complete with a TensorRT-LLM compile-time penalty, you are reading recycled content built on a claim that stopped being true in September 2025.

Scheduling and prefix reuse

This is the most useful technical distinction between vLLM and SGLang, and it maps directly onto workload shape.

vLLM uses automatic prefix caching, hash-block based and enabled by default since the V1 engine, designed so that the overhead is effectively nil even at a zero percent cache hit rate. It handles the common case very well: a single chain of conversation, a repeated system prompt, a long document reused across turns.

SGLang uses RadixAttention, which holds cached KV blocks in a radix tree with LRU-style eviction over the tree structure and matches by longest common prefix. The design target was many-branch programs: a single prompt that fans out into dozens of continuations, agent trees, self-consistency sampling, batched evaluation runs. If your traffic looks like that, tree-structured reuse finds sharing that block hashing does not.

SGLang’s second lever is its zero-overhead CPU scheduler, which overlaps the scheduler’s own Python work with GPU compute. This matters most on small models at high request rates, where scheduling overhead rather than matrix multiplication is the bottleneck. It is the main structural reason SGLang has historically won small-model throughput comparisons.

TensorRT-LLM competes on kernels rather than scheduling. Wide expert-parallel MoE, NVFP4 fused MoE kernels, and rack-scale NVL72 topologies are where its engineering goes. It also has one capability the others do not advertise: guided decoding composable with speculative decoding, since v1.1. Combining constrained output with speculation is historically fragile, and if you need strict JSON at speculative-decoding speed, that is a genuine differentiator.

Quantization and hardware fit

Two practical notes. bitsandbytes moved out-of-tree to a plugin in vLLM v0.28.0, which is a breaking change if your pipeline depends on it. And vLLM’s AMD support is narrower than its NVIDIA support at the format level: GPTQ, GGUF and FP8 W8A8 work, AWQ and Marlin do not.

On hardware, the split is stark and it settles more shortlists than throughput does. vLLM runs on NVIDIA, AMD ROCm, TPU, Intel Gaudi, Intel CPU and XPU, AWS Neuron, Huawei Ascend, and more through plugins. SGLang covers NVIDIA and AMD in production and added TPU support through SGL-JAX in mid-2026. TensorRT-LLM is NVIDIA-only.

The audited evidence for AMD viability is strong: in MLPerf v6.0 every MI355X and MI350X submission ran a PyTorch/ROCm stack, several declaring vLLM explicitly, and one AMD MI355X llama2-70b Interactive result at 73,608 tokens/second came in ahead of an 8-way B300 TensorRT submission at 70,724. If diversifying away from single-vendor GPU supply is anywhere in your three-year plan, that is the fact to plan around.

Verdict by profile

• Startup serving one product model, small team, no GPU specialist: vLLM. Widest model support, no compile step, largest body of community answers when something breaks at 2am.
• Agent product, RAG behind a fixed system prompt, or heavy multi-turn chat: SGLang. RadixAttention is built for exactly this traffic shape, and the scheduler overhead advantage compounds at high request rates on small models.
• Platform team serving many models across mixed accelerators: vLLM, orchestrated with llm-d if you are Kubernetes-native or NVIDIA Dynamo if you are not. Dynamo is engine-agnostic and appeared in MLPerf v6.0 submissions with both vLLM and TensorRT-LLM behind it.
• NVIDIA-committed enterprise, stable model set, hard latency SLA: TensorRT-LLM. The Server-scenario advantage in the audited data is real, and post-1.0 the operational cost of adopting it is far lower than its reputation implies.
• Research or frontier-model team: SGLang for day-0 availability, with vLLM as the fallback for anything long-tail.
• AMD in the roadmap: vLLM or SGLang. TensorRT-LLM is not a candidate.

Common pitfalls

• Citing a benchmark without a date and a version. Engine performance moved 9 percent on identical hardware in six months, and multiples on new models. A 2024 number is archaeology.
• Assuming TensorRT-LLM still needs an engine build. It has not since v1.0 in September 2025. Half the comparison content online is describing 0.x.
• Benchmarking without normalising speculative decoding. Acceptance length swings results enormously, which is why the serious continuous benchmark had to invent a synthetic acceptance mechanism to compare engines at all.
• Treating “fastest” as a property of the engine. Tensor and expert parallel degree, CUDA graph mode, MoE backend, KV cache dtype and chunked prefill budget each move results double digits, and each engine has a different optimum.
• Ignoring governance. All three are Apache-2.0, but neutral foundation stewardship, a VC-backed startup, and a single hardware vendor are three different long-run risk profiles. Pick with your eyes open.

Related reading

• BentoML vs KServe - the packaging and platform layer that sits above whichever engine you pick
• Build vs Buy RL Training Infrastructure - where inference engines fit inside an RLHF topology, and why the engine is a buy
• Hire an ML Engineer 2026 - the skills to screen for if you are staffing this stack in-house

Getting help

Serving-engine choices get sticky fast: once autoscaling policy, quantization pipeline, benchmarking harness and on-call runbooks are built around one engine, moving costs real engineering months. Our Model Design & Selection sprint runs the benchmark on your model, your prompt distribution and your latency target rather than someone else’s, and our ML Platform Engineering engagements implement the serving, autoscaling and monitoring layers around the result. If the stack is already live and the problem is cost per token, an ML Architecture Review finds where the GPU budget is leaking.

Book a free scope call.