What Was Released?

Chinese AI lab DeepSeek has open-sourced DeepInfer, the inference framework it says powers its own large-scale production services. The code went up on GitHub on Tuesday under the permissive Apache 2.0 license, alongside a companion repository called deepseek-recipe, its token encoding and decoding component that matches the official API, AI Weekly reports.

DeepSeek announced the release in a four-part thread on X that ended with the line: “Open models. Open kernels. Open inference.” The thread opens: “Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”

The first release targets DeepSeek-V4.1-Flash, the model the lab put on Hugging Face under the MIT license on October 18, and runs it in offline mode. The repository gathered more than 11,000 GitHub stars within its first day.

Illustration of DeepSeek's DeepInfer open-source inference framework

What Is Inside

An inference engine is the software that serves an already-trained model to users: it splits the model across chips, batches requests, and decides how fast answers come back. That is a big part of what an AI company pays per answer, and DeepSeek’s famously low prices have always rested on how well it does this. Until now, the full engine stayed closed.

DeepInfer supports two chip families: NVIDIA SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2). The Huawei support is notable: it amounts to the first first-party production endorsement of Huawei’s latest AI silicon from a frontier lab, and DeepSeek said last week it plans to pre-train on Ascend hardware by the end of 2026.

The feature list bundles DeepSeek’s recent systems research into one operational package. Custom kernels include FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, with TileLang in the mix. Parallelism covers attention data parallelism with balanced context parallelism plus mixture-of-experts expert parallelism with expert load balancing. Decoding uses DSpark draft tokens with confidence-scheduled verification, and generation can be constrained to regex, JSON schema, or EBNF grammar. It also offers a batch-invariant inference mode with deterministic sampling, so a request returns identical outputs for identical inputs regardless of what else is in the batch.

DeepSeek’s own production figures, reported from the repo’s Q&A material and not independently tested: 155 trillion tokens per day per cluster, a 96% input cache hit rate, average time-to-first-token around 400 milliseconds, and 230 tokens per second (up to 300 for coding), Tesla News Wire reports.

A Notable Reversal

DeepInfer is a genuine turnaround. In a note in April 2025, DeepSeek said it had considered releasing its full inference engine but decided against it: the code was a heavily customized early fork of vLLM, “tightly coupled” with its internal cluster tools, and its small team lacked the bandwidth to maintain a big open-source project. It promised to contribute pieces to existing projects instead. The new README still credits vLLM as the base DeepInfer was “originally built primarily on,” with ideas from SGLang and NVIDIA’s TensorRT-LLM.

The release is not a complete production system. The README lists what a full service still needs and this release leaves out: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Serving the way DeepSeek does is a separate engineering project, as Golden Era of Super Intelligence notes.

Illustration related to DeepSeek DeepInfer serving V4.1-Flash at scale

Why It Matters

This is the part of the AI stack most labs never publish. Training code gets open-sourced; the serving machinery that turns tokens into revenue at a profit rarely does. By releasing Apache 2.0 code that anyone can use, change, and sell commercially, DeepSeek directly threatens managed inference pricing at providers like Fireworks, Together, and Groq.

It also deepens the open-model pressure on Western labs. As one widely liked reply to DeepSeek’s announcement put it: OpenAI, “supposed to be open for the benefit of humanity, ended up staying closed.” Whether or not that critique is fair, the gap between the two approaches is now concrete, downloadable, and star-counted: one company’s internal serving stack is now everybody’s.

Sources