DeepSeek open-sources DeepInfer, the engine it uses to serve its models

DeepSeek released DeepInfer, the inference framework behind its production services, as open source on 20 October 2026. It announced the move on X and published the code on GitHub under the Apache 2.0 licence. A companion library, deepseek-recipe, also went public; written in Rust and Python, it converts API requests into model inputs and handles outputs the way DeepSeek's own API does. It first appeared quietly in September and was updated on Tuesday. DeepSeek's deployment Q&A is the source of the throughput and revenue figures below, so they are the company's claims, not independent measurements.

For now the engine supports a single model, DeepSeek-V4.1-Flash, in the updated version, build 1017, released the weekend before, so no other DeepSeek model can run on it yet. It targets Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 NPUs; Hopper-generation and older Nvidia chips are not supported. It builds on DeepSeek's earlier FlashMLA, DeepGEMM and DeepEP kernel libraries and adds speculative decoding, asynchronous scheduling and structured outputs. Its speculative decoding, called DSpark, drafts five tokens at a time with a confidence head and is estimated to add 4 to 7 percent throughput at production load. Deterministic sampling and batch-invariant inference work only on Nvidia, and the published benchmarks are DeepSeek's own, run on Nvidia hardware, with none reported for Huawei's chips.

“From a market messaging perspective, they want to be perceived as the leader in the space, so anything they can do to put stuff out there that says we're exhibiting our leadership is good for them, and it's good for China in general”

Jack Gold, president and analyst, J. Gold Associates, to TechTarget · techtarget.com ↗

DeepSeek says one cluster serves 155 trillion tokens a day and that off-peak revenue works out to $38 per GPU-hour. A prefill technique that replays only the last 128 tokens of a prompt is said to nearly halve prefill compute. Outlets disagree on the hardware needed: SiliconANGLE describes a smallest efficient deployment of 64 prefill and 64 decode GPUs, while The New Stack says the published measurements used four SM100 GPUs for prefill and 64 for decoding.

The release is partial. It covers offline inference only, leaving out the KV cache system, prefill and decode disaggregation and load balancing across replicas, though DeepSeek published a Zhihu post explaining how the last two work. It says code for future models will go into the same repository. In April 2025 it had declined to release its full engine, an early fork of vLLM tied to internal cluster tools, promising components instead; the new README names vLLM as the foundation.

DeepSeek frames the release as sharing its production stack to help the community lower compute costs, cut energy use and make AI cheaper, and says serving a huge user base at reasonable cost is the core problem such a framework must solve. Omdia's Lian Jye Su, speaking to TechTarget, says soaring token consumption makes faster, cheaper turnaround valuable, and reads the Nvidia dependence as a sign Huawei still lags in some areas. J. Gold Associates president Jack Gold sees a leadership signal that helps DeepSeek, China and Huawei's chip reputation.

The release continues a run of component launches, after FlashMLA, DeepGEMM and DeepEP, and the hardware angle has a recent history. On 30 September DeepSeek brought its TileLang kernel language to Huawei chips with Huawei's help, and Bloomberg reported plans for at least 160,000 Ascend 950DT chips at an Inner Mongolia data centre. The release also landed a day after the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products, a ban China called an attempt to politicise technology.