Local AI Advisor
All posts

DeepSeek open-sourced its inference engine — here's what it changes for you

DeepSeek open-sourced DeepInfer, the engine behind its production AI service. What it does, what it leaves out, and why your GPU can't run it yet.

By Ray, Your Local AI Advisor6 min readai-newslocal-llmannouncements

DeepSeek just did something it said it wouldn't do: it published the software that runs its own AI service. On October 20, 2026, the company put DeepInfer — "the high-performance inference framework that powers DeepSeek's large-scale production services" — on GitHub under the Apache 2.0 license. One takeaway before we go further: you cannot run it on your own machine, and the company is honest about that. What you can do is read it, learn why DeepSeek's API is so cheap, and watch its ideas flow into the engines that run on home hardware.

Flat illustration of a server rack streaming glowing token blocks through stacked GPU chips to a laptop, phone and tablet

What DeepInfer actually is

An inference engine is the software that takes a finished, trained model and serves answers to millions of people fast and cheaply. It splits the model across chips, batches requests together, and decides how quickly answers come back. A big share of what an AI company spends per answer lives here — which is exactly why DeepSeek's version is interesting. The company says it serves DeepSeek-V4.1-Flash with DeepInfer at a scale of 155 trillion tokens per day per cluster.

The release itself is narrow by design. The README supports exactly one model — DeepSeek-V4.1-Flash, with the default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloading into the Hugging Face cache on first run — and runs offline inference only. No online serving, no streaming multi-user stack. DeepSeek even lists the pieces a complete service still needs and admits this release leaves them out: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas.

Hardware support is just as narrow: NVIDIA SM100 GPUs (the Blackwell generation, with CUDA 13) and Huawei Ascend 950 NPUs (with CANN 9.2). These are data-center chips. Your RTX card, your Mac, your gaming PC are not on the list. Prerequisites are Linux, GCC 13 or newer, and Python 3.12.

The turnaround is the story here. In a note in April 2025, DeepSeek said it had considered releasing its full inference engine and decided against it: the code was a heavily customized early fork of vLLM, tightly coupled to its internal cluster tools, and its small team lacked the bandwidth to maintain a big open-source project. It promised to contribute pieces to existing projects instead. The new README still credits vLLM as the base DeepInfer was "originally built primarily on," with ideas drawn from SGLang and NVIDIA's TensorRT-LLM. Something changed between then and now — and on its official X account, DeepSeek said it is sharing the stack "to help the community lower compute costs, reduce energy use, and make AI more affordable."

The speed tricks, in plain terms

DeepInfer's feature list reads like a greatest-hits album of modern serving, and each trick maps to something this blog has covered on home hardware:

  • Draft tokens (DSpark): the engine guesses a few tokens ahead and checks them in one pass instead of writing one token at a time. This is the speculative-decoding family — the same idea behind TensorFold, which we covered earlier this month — with a default maximum draft length of 5 and "confidence-scheduled verification" that spends checking effort where the guesses look shakiest.
  • SWA bounded replay: during prefill (reading your prompt), the decoder replays only a bounded suffix — 140 tokens by default for V4.1 — instead of the whole input, cutting the recomputation each request needs.
  • Custom kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels — the kind of chip-specific fast paths that self-tuning engines like Magnitude generate for your exact hardware.
  • Expert parallelism: attention data parallelism and MoE expert parallelism with load balancing (MoE EP + EPLB) to spread the model's mixture-of-experts layers across many chips.
  • Asynchronous scheduling: overlapping CPU batch scheduling with device execution, on by default, so the scheduler never blocks the GPUs.
  • Batch-invariant inference and deterministic sampling (NVIDIA only): the same request gets the same result no matter what else happens to be in the batch, and identical inputs with identical sampling settings produce identical outputs. For anyone running agents, that determinism matters.
  • Structured outputs: generation constrained to a regex, JSON schema, or EBNF grammar — so an agent gets valid JSON instead of almost-valid JSON.

Alongside the engine, DeepSeek released deepseek-recipe: Rust libraries with Python bindings under the MIT license that convert requests in OpenAI or Anthropic formats into DeepSeek's internal format and parse reasoning content, tool calls, and stop sequences back out.

Text-free diagram of the DeepInfer pipeline in three stages: prefill reads the prompt, draft tokens are guessed and verified in parallel during decode, and output comes back in a fixed structured shape

DeepSeek's own numbers — read them carefully

Bar chart of DeepSeek's self-reported production figures for V4.1-Flash: 155 trillion tokens per day per cluster, 96 percent input cache hit rate, about 400 milliseconds average time to first token, and 230 tokens per second decode, up to 300 for coding

DeepSeek reports that its production setup serves 155 trillion tokens per day per cluster, with a 96% input cache hit rate, an average time to first token of about 400 milliseconds, and 230 tokens per second decode throughput (up to 300 for coding). Treat these as company numbers, not independent measurements: the README's own benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode, with no comparison against vLLM or SGLang.

Still, the chart is worth studying, because it shows where cheap inference comes from. The 96% cache hit rate is doing enormous work: when prompts repeat (system prompts, agent loops, prefixes), the engine mostly reuses cached work instead of recomputing it. And the draft-token decoding plus custom kernels squeeze more output per second out of each GPU. Cost per answer is mostly a function of these two things — cache hits and decode speed — not of the model being magically cheaper to run.

So can you run this on your own machine?

No. Not today. The hardware wall (SM100 or Ascend 950 only), the one-model limit, offline-only operation, and the missing service pieces (KV cache system, disaggregation, load balancing) make DeepInfer a research artifact for you, not a tool you install on your gaming PC or Mac.

But "research artifact you can read" is still a big deal, for two reasons:

  1. Apache 2.0 means the ideas are portable. Anyone can fork DeepInfer, study the kernels, and port the techniques. DeepInfer's bag of tricks is a close cousin of what already runs locally — draft-token decoding like TensorFold's, custom decode kernels like DwarfStar's, hardware-specific optimization like Magnitude's. The techniques that prove themselves in DeepInfer tend to land in vLLM, SGLang, and llama.cpp within months, and from there onto your hardware. Check what fits your machine on our /check page.

  2. It teaches you to read serving claims like an engineer. When a provider says "cheap and fast," ask the two DeepInfer questions: what's the cache hit rate, and what does decode actually cost per token? The same questions sharpen how you think about your own local setup — prompt caching in llama.cpp and Ollama is your version of the same game.

The release landed the day after the US government ordered federal agencies to remove DeepSeek models from their systems. Coincidence or message, the code is real, the license is permissive, and the techniques are now public. Your RTX card can't run DeepInfer — but the next generation of the engines it can run just got a new textbook to study.

Sources