Local AI Advisor
All posts

llm-d 1.0: the open serving stack behind the 11.5x cheaper self-hosted AI claim

llm-d 1.0 ships the three serving pieces DeepSeek's open engine left out: free, vendor-neutral, production-ready. IBM Research reports 11.5x lower cost.

By Ray, Your Local AI Advisor6 min readai-newslocal-llmannouncements

Yesterday we covered DeepSeek's DeepInfer — the open-sourced engine behind its production AI service — and noted something the company admits in its own README: an engine alone is not a serving stack. DeepSeek listed three pieces a real service still needs that its release does not ship: a KV cache system, prefill/decode disaggregation, and load balancing across replicas. One day later, all three have been shipped. Free, open source, and vendor-neutral. One takeaway before the details: this is the second half of yesterday's story, and the half that decides what self-hosted AI actually costs.

Flat illustration of GPU server racks streaming into a central routing hub that serves a ring of small AI agent bots

What llm-d actually is

llm-d is not an inference engine. Engines like vLLM and SGLang are the software that takes a finished model and generates tokens on one machine or one set of GPUs. llm-d sits above them: a Kubernetes-native layer that takes a whole cluster of vLLM or SGLang servers and turns them into a single coordinated pool. It routes requests cache-aware, offloads KV cache across memory tiers, splits prompt-reading from answer-writing onto different machines, and balances load across parallel copies of a model.

The project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA. It is governed by the Cloud Native Computing Foundation as a sandbox project under the Apache 2.0 license, with more than 500 contributors from over 25 organizations. Version 1.0 was announced on October 20, 2026, at the PyTorch Conference in San Jose.

The 1.0 part is not about raw speed — prefix-aware routing, KV cache offloading, and prefill/decode disaggregation already existed. What 1.0 adds is production machinery: multi-tenant quality-of-service controls so chat users and bursty agents share one GPU fleet without starving each other, fairness-aware flow control, a highly available router for zero-downtime upgrades, session-aware routing that keeps multi-turn conversations next to their cached state, and a stable v1 API.

The three missing pieces, delivered

DeepInfer's missing list maps onto llm-d 1.0 almost line for line:

  • KV cache system → a prefix-aware router reads each replica's KV-cache state and sends each request to the replica that already holds its prompt prefix, so long shared prefixes (system prompts, tool definitions, conversation history) are looked up instead of recomputed. Offload tiers span GPU memory, CPU memory, SSD, and managed Lustre.
  • Prefill/decode disaggregation → prefill (reading your prompt, compute-heavy) and decode (writing the answer, memory-heavy) run on separate machine sets, so neither wastes the other's resource. This same DisaggregatedSet component underpins the zero-downtime upgrades.
  • Load balancing across replicas → prefix- and load-aware routing with flow control that can queue or shed low-priority agent traffic so interactive requests stay responsive — the priority bands Mistral's inference engineers specifically called out for their commercial tiers.

The engine made tokens cheap to produce; this layer makes them cheap to serve to thousands of users and agents at once. Google's suggested path for existing setups: put prefix-aware routing in front of a vLLM deployment you already have, then add flow control, KV cache offload, and disaggregation as you grow.

The numbers — read them carefully

IBM Research published the headline figures. Every one of them is vendor self-reported, not independently audited:

  • On a quantum error correction research workload, open frontier models served with llm-d came out 11.5 times cheaper than an all-proprietary configuration, at a blended serving cost of about $0.11 per million tokens, with a prefix-cache hit rate of roughly 98%. At that hit rate, nearly all of the most expensive computation (prefill) becomes a cache lookup.
  • In a separate agentic-workload demo, llm-d deployed GLM 5.2 — an open-weight mixture-of-experts model of roughly 753 billion parameters — on widely deployed NVIDIA GPUs, delivering over 6.6 million output tokens per minute at peak and serving up to 3,000 concurrent coding agents with zero preemptions. At current cloud rental rates, the team says that was 5 to 10 times cheaper per token than equivalent commercial API pricing, with the largest savings on input-heavy agent traffic.
  • Google demonstrated 1,000 sandboxed agents hitting a single llm-d endpoint across vLLM, SGLang, and a mixed fleet of TPUs and GPUs, with cache-aware routing keeping each agent's steps on the replica holding its context. On Ironwood TPUs, prefix-aware routing plus host memory offload cut median time-to-first-token by more than 85% and raised throughput 2.4x versus a plain Kubernetes service.

Bar chart comparing IBM Research's reported blended cost per million tokens: about $0.11 for self-hosted open models with llm-d versus an estimated $1.27 for an all-proprietary stack

The chart above is derived: $1.27 comes from $0.11 times the reported 11.5 ratio. Treat it as a direction, not a quote. The 11.5x figure was measured on one long-context research workload where prompts repeat; short single-shot chats would see far less.

The adoption list is the other signal: JetBrains says llm-d serves a significant share of the traffic behind its free Junie Lite coding-agent plan, DigitalOcean runs serverless inference on it, Mistral and Roblox describe building their serving layers on the project, Google is integrating 1.0 into Google Kubernetes Engine across its TPU and NVIDIA GPU portfolios, and AMD is bringing it to its Instinct accelerators.

Can you run this on your own machine?

Honestly: no — not on one GPU. llm-d is Kubernetes-first and assumes a fleet of accelerators. If your setup is a single card and an Ollama instance, this is not software you install this weekend.

But it matters for local AI people:

  1. The economics argument flipped. "Just use the API" was the safe default because self-serving was finicky and expensive. IBM's numbers say that on agent-shaped workloads, serving open models yourself is now 5 to 10 times cheaper per token than equivalent commercial API pricing. The savings live in long, repetitive agent traffic — exactly where costs explode for anyone running agents for a team.
  2. The savings layer stays shared. CNCF governance with the Apache 2.0 license means no single company can take it proprietary or change its direction alone. If a founder walks away, the remaining 500-plus contributors keep shipping under the same rules.
  3. The techniques trickle down. Prefix-aware routing is prompt caching at fleet scale; disaggregation is the same instinct as splitting work across machines you already own — the enterprise-grade version of the idea behind NVIDIA's PAIR, which we covered on October 6. What proves itself here lands in the engines that run on home hardware.

A footnote from the same week: on October 21, Ant Group and the vLLM community jointly open-sourced Humming, a quantized inference kernel library that reportedly beats vLLM Marlin, FlashInfer, and DeepGEMM across typical workloads. The engines keep getting faster while the serving layer gets smarter — and both are landing in the open.

If you run a team, a dev shop, or a serious homelab, price out serving an open model yourself instead of renting API tokens. Check what fits the hardware you own on our /check page — then imagine that math with 98% of your prefill served from cache.

Sources