On October 20, DeepSeek announced that it has open-sourced DeepInfer, a large-scale cloud inference framework. The framework already powers large-scale online inference for DeepSeek V4.1 Flash.

According to figures released by DeepSeek, a single cluster running on the system currently serves 155 trillion tokens per day. At off-peak pricing, that works out to $38 per card per hour in revenue.

The release goes beyond the inference framework itself. DeepSeek also open-sourced deepseek-recipe, a production-grade token encoding and decoding component that converts between mainstream large-model API protocols and model tokens. Its handling logic is fully aligned with the official API, the aim being to give third-party deployments an API experience as close as possible to DeepSeek's own service.

Repositories: DeepInfer: GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models; deepseek-recipe: GitHub - deepseek-ai/deepseek-recipe

This is not, however, a complete release of DeepSeek's entire online inference infrastructure. The company stated plainly that its KVCache management and load-balancing components are tightly coupled to internal infrastructure, and that the relevant code has not yet been fully open-sourced. What this disclosure focuses on is the overall architecture of the V4.1 Flash inference system, its key implementation choices, and the design reasoning behind these components in real large-scale online service.

Eight Cards for Prefill, 64 to 128 for Decode

DeepSeek says the V4.1 Flash inference system was built around three core goals from the outset: compatibility with compute chips from different vendors; organizing Prefill and Decode around commonly available hardware; and keeping KVCache alive for more than 72 hours to cut down on repeated computation.

In practice, standard eight-card servers handle Prefill, while supernodes of 64 to 128 cards handle Decode. The two are no longer treated as the same kind of compute workload; each gets its own compute, communication, and scheduling strategy based on its characteristics.

The architecture carries over parts of the design used in the DeepSeek-V3 and R1 era, but goes further by deploying the vision encoder, Prefill, and Decode separately, so all three stages can be resourced independently and tuned for time-to-first-token (TTFT) and time-between-tokens (TBT) respectively. On the mixture-of-experts (MoE) side, Prefill typically uses two- to eight-way expert parallelism alongside data parallelism, while Decode scales out to 64- to 128-way Wide-EP, using larger batch sizes to improve GPU matrix compute efficiency.

One clear shift is that inference system optimization increasingly revolves around how to reduce idle resources across the entire cluster.

Once Prefill and Decode are split apart, the demand ratio between the two resource pools shifts in real time with online traffic. Fix the ratio in advance and you end up with queues at some hours and idle machines of the other type at others. To address this, DeepSeek added automatic scaling at the cluster layer: the Prefill side determines how many instances it needs based on the number of uncached tokens, while the Decode side estimates demand from the actual number of compute requests in flight. The scaling process has upper and lower bounds as well as rate limits on adjustments.

To curb the waste caused by long-tail requests in large-EP deployments, a Decode instance that receives a shutdown notice does not exit immediately but enters a one-minute grace period. If requests remain unfinished when the grace period ends, the system forwards those requests along with their KVCache to other instances in one shot to continue decoding. Deployments also hold back redundant quota: some machines sit in a Ready state without being assigned work, serving as inventory for sudden scale-ups, and the number of online instances is always kept above theoretical demand. When quota utilization exceeds 0.8, the system blocks scale-down outright.

On fault tolerance, DeepSeek did not try to squeeze instance restart times down to the bare minimum. Restarts after an error still take minutes, with the system relying instead on a combination of redundant quota and request-level rerouting.

In large-scale EP deployments, all GPUs in an instance form a single communication domain, so a failure in any one card can take the whole instance out of service. On restart, to keep using the remaining healthy GPUs, the platform recounts available cards and resets the EP size based on how many healthy GPUs it actually gets.

Data parallelism, expert parallelism, Prefill, Decode, and vision encoding each need a different load balancer, so DeepSeek built four: one each for Prefill, Decode, the Vision Encoder, and Expert-Parallel.

Prefill, for instance, sorts requests into four buckets by input length — under 16K, 16K to 64K, 64K to 256K, and above 256K — and allocates engines dynamically based on actual compute load. Decode watches both request count and total KVCache length, placing long and short requests on different cards to reduce the throughput lost when some devices sit waiting.

Expert parallelism is adjusted dynamically as well. Beyond its resident experts, each card hosts additional redundant experts as replicas of popular ones, reallocated according to each expert's actual load over a recent window. Online figures published by DeepSeek show that the load gap between compute cards in Prefill has been narrowed to a low level.

100 PiB of SSD Dedicated to KVCache, With Chat Traffic Held for Seven Days

The other component that takes up a large share of DeepInfer is KVCache.

As context windows grow longer, and with agents continuously carrying along conversation history and tool-call records, the cost of repeated Prefill rises fast. DeepSeek's answer is to turn KVCache from an ordinary GPU-memory cache into a cluster-level shared data layer.

It splits into Block KVCache and State KVCache. Block KVCache is paged in blocks of 128 tokens, with persistent data written to a dedicated 3FS cluster. The system separately maintains an SWA KVCache: each of the first 20 layers holds a 134-token ring, of which the window length is 128 tokens, with 6 tokens reserved for speculative rollback.

Block KVCache and State KVCache in active use sit in inference card memory, managed by the Prefill and Decode instances. SWA KVCache lives in the host memory of the machine that produced it, is never written to 3FS, and is kept as a single copy; cross-machine hits are pulled over a 400G VLAN.

Rather than the common multi-tier cache approach, DeepSeek puts persistent Block KVCache straight into a standalone storage cluster. That way any Prefill machine can hit any prefix, removing the need for prefix-affinity routing and avoiding conflicts between cache routing and load balancing. Storage nodes and compute nodes can also scale independently, and a compute node failure does not take persistent KVCache with it — DeepSeek notes that compute node failure rates run an order of magnitude higher than those of storage nodes.

3FS currently provides roughly 100 PiB of SSD capacity for KVCache. The cache time-to-live is 3 days for agent traffic and 7 days for chatbot traffic. The latter shows a more pronounced long-tail hit pattern: a small amount of KVCache may be read back very frequently, so deleting it too early does more damage to the overall hit rate. In DeepSeek's experience, cutting chatbot KVCache TTL to around 3 days lowers the overall hit rate by 0.5% to 1%.

On bandwidth, 3FS has been tuned specifically for random reads and writes of KVCache blocks at roughly 112 KiB granularity, reaching peak aggregate read bandwidth of 10 TiB/s and peak aggregate write bandwidth of about 850 GiB/s.

DeepSeek also walked through the KVCache hit path. Block KVCache is written to disk by Decode instances, so data can be reused by other requests regardless of whether it came from Prefill or Decode. The flow has three stages, only the first of which sits on the request's critical path: the Entrypoint first performs prefix matching via a hash chain in the local kvcache-store acting in the CHECK role, a metadata-only lookup, but one whose hit length must be known before routing, so it cannot be hidden behind other computation. A separate process then asynchronously moves the KVCache from 3FS into host shared memory, and finally the Engine thread moves it from shared memory into HBM.

The goal is to overlap layer-by-layer host-to-device transfers with forward computation. Because the CUDA Copy Engine performs poorly on fine-grained KVCache block copies, DeepSeek switched to GDR, letting the RDMA NIC saturate PCIe bandwidth without consuming SMs.

On KVCache management, V4.1 Flash also reverses one design decision from the V4 era.

Previously the system wrote the KVCache for sliding window attention (SWA) to persistent storage so it could be restored exactly when needed. In production, though, DeepSeek found that this kind of cache is typically reused only within the few minutes of a single active session, unlike Block KVCache, which can be hit again days later. Keeping minute-scale data on large amounts of SSD for the long term would consume nearly half of persistent cache capacity without delivering a matching return in hit rate.

V4.1 therefore adopts SWA Bounded Replay: SWA KVCache is no longer written to SSD for the long term, and when it is missing, only the most recent 128 tokens are recomputed, with the recomputation scope capped to a fixed window. Recomputation cost no longer climbs with full context length and model depth, becoming roughly a fixed overhead instead. DeepSeek says that after applying this method on the decoder side, each Prefill only needs to recompute the final 128 tokens of the prompt, cutting total Prefill computation nearly in half compared with the earlier approach of precisely rebuilding large amounts of state layer by layer.

Speculative Decoding No Longer Verifies a Fixed Five Tokens

DeepInfer also integrates DSpark, DeepSeek's speculative decoding framework.

Conventional speculative decoding generates a fixed number of candidate tokens at a time and hands them all to the main model for verification. The problem is that candidate tokens at different positions do not share the same acceptance probability, so verifying a fixed five can burn compute on positions that were never likely to be accepted.

DSpark adds confidence prediction, deciding how many candidates to verify based on each one's expected acceptance rate. In actual online deployment the system still generates five candidate tokens at a time, but picks the subset with the highest expected payoff for verification, guided by hardware throughput curves and predicted acceptance rates. The result: variable-length verification raised throughput by 6% in real online deployment compared with verifying a fixed five tokens.

Decode scheduling also runs GPU and CPU work in an asynchronous pipeline. In the traditional flow, the GPU may sit waiting while the CPU handles the next round of scheduling. DeepInfer has the CPU complete the next round of scheduling while the GPU runs the current round's forward pass and sampling, overlapping computation and scheduling as much as possible to keep the GPU busy.

Even the API-to-Token Layer Is Open Source

Alongside the core inference engine, the release includes deepseek-recipe, a production-grade component that has drawn little attention until now.

It handles conversion between mainstream large-model API protocols and the token sequences DeepSeek's models actually consume. It currently supports three APIs — Chat Completions, Responses, and Messages — normalizing the different protocols into an internal conversation format, then generating the corresponding prompt and token sequence for the specific model version. Request conversion, prompt encoding, and output parsing are implemented primarily in Rust, with Python bindings provided.

DeepSeek paid particular attention to special-token injection. Even if a user's input contains <｜User｜>, <｜image｜>, or similar tool-call tags, these are not interpreted as control tokens carrying special semantics; they are encoded as ordinary text. That prevents special strings in user input from acquiring control semantics and affecting model inference behavior.

For long-context scenarios, DeepSeek exploits the known boundaries of special tokens to tokenize separate plain-text segments in parallel and then reassemble them, reducing the impact of very long inputs on CPU load and time-to-first-token. On streaming output, DeepInfer maintains a cross-chunk streaming parser state machine, so even when a single UTF-8 character, tool-call argument, or structural tag is split across multiple data chunks, parsing waits until the data is complete while still returning confirmed portions as early as possible.