DeepSeek V4 Pro DSpark: the model isn't ready, the architecture is

From July 14 to July 16 we ran a public experiment on umans.ai: DeepSeek V4 Pro with DSpark speculative decoding, open to community volunteers on rotating Labs seats.

We were there for the architecture. DeepSeek V4 changes two things that decide how many people a GPU can serve at once, and we expect the rest of the open-weight field to adopt both. We wanted our hands on them, on real traffic, before that happens.

Attention that scales almost linearly

Attention is the part of a transformer that scales badly. Every new token attends to every token before it, so the work per token grows with the length of the conversation, and the work over a session grows with the square of it. The KV cache grows alongside, one entry per token per layer, for as long as the conversation runs.

There are two places to fight this: the serving stack, or the model architecture itself. The serving stack can make a large cache cheaper to hold: paging, quantization, shared prefixes, prefill and decode disaggregation. The architecture changes how big the cache is in the first place. DeepSeek V4 does that by changing what each token reads: it attends to a compressed representation of the past instead of to every token in it. The product that used to be token × token becomes token × rep(token), and a bounded right-hand side turns a quadratic curve into a nearly linear one. Two mechanisms do the compressing (DeepSeek-V4 technical report, §2.3):

• Compressed Sparse Attention. Collapse every m tokens into a single KV entry, then use a lightweight indexer to attend to only the k most relevant of those, with a sliding window alongside for local detail. Compression and sparsity stack.
• Heavily Compressed Attention. Compress far harder, then attend densely over the result, because there is so little left to attend to.

At a 1M-token context that puts V4’s KV cache one to two orders of magnitude below a conventional BF16 baseline. Against DeepSeek’s own previous generation at that length, V4 Pro runs at 27% of the single-token FLOPs and 10% of the KV cache of V3.2, and V4 Flash at 10% and 7%.

Those are two different constraints, and you need both. KV cache sets the ceiling on how many sessions can be live on a GPU at once. FLOPs decide whether you can actually serve them fast enough to be worth having. A model can fail either test: too much cache and the sessions do not fit, too much compute per token and the ones that fit crawl. Xiaomi’s MiMo team puts the second half plainly: once memory is filled by KVCache, batch size cannot expand, GPU compute units are not saturated, and decode throughput is limited.

It also pays off more than the raw numbers suggest, because agentic sessions are bursty. A coding session decodes for a few seconds, then goes quiet while tools run, a test suite finishes, or you read the diff. Only an actively decoding session needs its KV in VRAM, so serving keeps it in tiers: VRAM for what is decoding right now, host DRAM below that, DRAM plus SSD below that again. Sessions are evicted downward when they go quiet and hauled back up when their next turn arrives, continuously. Sparse attention pushes the same way inside a single step, since a decode that only reads selected blocks can leave the rest a tier down.

So the ceiling is two numbers: how many decoding sessions fit in the fast tier, and how quickly a sleeping one is rehydrated when it wakes. Shrinking the cache moves both, and the second is latency your users feel directly, sitting between their next message and its first token. It builds on the long-context serving math from host-claude-code and GLM-5 vs Kimi-K2.5.

The whole field is heading this way. A recent survey of cross-datacenter serving (Prefill-as-a-Service, Qin et al., April 2026) measures today’s hybrid-attention models against their full-attention predecessors and finds KV cache reductions of 4x to 36x: 13x for MiMo-V2-Flash over MiniMax-M2.5, 4x for Qwen3.5-397B over Qwen3-235B, roughly 36x for Ring-2.5-1T. The footprint has fallen far enough to change what is possible at the systems layer, and they take it to the extreme: run prefill in a different datacenter and ship the cache back over commodity Ethernet. Same bet as ours, several tiers further out.

That is the memory half. The speed half is DSpark.

The frontier

Here is the plot that made us want to run this experiment. It is Figure 7 from DeepSeek’s DSpark paper, measured on their own live production traffic.

The x-axis is tokens per second per user, how fast one session feels. The y-axis is aggregate throughput per GPU, how many people it can serve at the same time. Those two fight each other on every serving stack ever built. A GPU generating tokens spends most of its time waiting on memory, so you extract work from it by batching: run many users through the same forward pass and amortize the weight reads. Bigger batch, more total throughput, and each user waits a little longer per token.

The curve is the constraint. Where you sit on it is a policy choice. Every capacity lever a serving stack has picks a point on it: concurrency limits, priority tiers, a separate lane for background work. They all decide who gets served how fast when the curve is too small for everybody. That is allocation, and allocation is zero sum.

Pushing the curve outward gives everybody something. At a fixed 35 tokens per second per user, DSpark carries 52% more aggregate throughput than the configuration DeepSeek ran before it. At matched throughput, each user’s tokens arrive 57% to 78% faster.

The paper also marks a 406% figure at a stricter 50 tok/s/user target, and is careful about it: at that interactivity level the old configuration collapses to a very small batch, so the ratio inflates. DeepSeek reads it as the frontier reaching a level the baseline could not serve at all. That reading is the useful one for us, because the regime where the baseline collapses is our peak hours.

Speculative decoding, briefly

Generating a token requires a full forward pass, and that pass is dominated by reading weights out of memory. One token’s worth of arithmetic comes nowhere near filling the tensor cores while that read happens, so the machine spends the decode phase largely waiting.

Speculative decoding fills the gap with a bet. A small, cheap draft model proposes a block of candidate tokens. The real model verifies the whole block in a single forward pass, because verification parallelizes in the way generation does not: checking twelve tokens costs roughly what checking one costs, since the expensive part was reading the weights either way. Whatever prefix the big model agrees with is accepted, plus a bonus token it generates itself. The rest is discarded and the cycle repeats.

The acceptance rule makes this lossless. It is rejection sampling, constructed so that accepted tokens follow the target model’s distribution exactly. Any token the big model would not have produced is thrown away and replaced with one it would. DeepSeek says it plainly on the release: DeepSeek-V4-Pro-DSpark is not a new model. It is the same checkpoint with an additional speculative decoding module attached. We have turned down fast checkpoints before for degrading the tokens they produced, and this class of acceleration removes that question entirely.

The catch is that rejected tokens cost real compute. Under light load that compute would have idled anyway, so the waste is free and you should bet as much as you can afford. Under heavy load, every token you verify and then reject occupied batch capacity another user needed.

That single fact explains the state of the art. DeepSeek’s production configuration before DSpark was MTP-1: one drafted token per cycle. Longer draft blocks help an individual user perfectly well, but a static longer block, in the paper’s words, strictly degrades aggregate throughput under high concurrency due to excessive verification overhead. Everyone serving at scale has been leaving single-user speed on the table, because the static version of this trade takes capacity from everybody to give speed to one.

What DSpark adds

Two mechanisms, and only the second one moves the frontier.

1. Draft better: a little autoregression goes a long way

Drafters come in two families. Autoregressive drafters generate the draft one token at a time, conditioning each on the last, which is good quality but forces tiny blocks and shallow networks because drafting time grows with block size. Parallel drafters emit the entire block in one forward pass, so cost barely depends on block size, which buys a deep drafter and a long block.

Parallel drafting has one structural flaw: each position is predicted independently, so no position knows what the others chose. If the continuation could plausibly be “of course” or “no problem”, an independent drafter can happily emit “of problem”, because position 1 averages over all futures while position 2 averages over all pasts. Acceptance decays down the block.

DSpark keeps the heavy backbone fully parallel and bolts on a tiny sequential head that walks the block left to right, nudging each position’s logits based on what the previous position actually sampled. Once position 1 has sampled “of”, it boosts “course” and suppresses “problem”.

2. Verify smarter: a throttle that costs the user nothing

Longer draft blocks on their own would make the throughput problem worse. This is the half that fixes it.

DSpark attaches a confidence head that outputs, for each draft position, the probability that this token survives verification given that everything before it survived. Multiply along the block and you get the survival probability of every prefix length. Those numbers feed an arithmetic decision rather than a ranking, so raw confidence (overconfident by 3% to 8% expected calibration error, as neural confidence always is) gets a per-position temperature fitted on held-out data to bring it to about 1%.

Then the hardware-aware scheduler uses them. Profile the engine once at startup to learn steps per second as a function of batch size. Every step, expected system throughput is expected accepted tokens times the speed at that batch size. Sort every candidate token from every active request by survival probability, admit them one at a time, and stop the moment the product stops rising.

The bottom row is the one to look at. The flat blue line at 2 is the old configuration: a static budget, at every load level, forever. The green is DSpark: roughly 5 to 6 tokens per request when the system is quiet, decaying toward 3 as concurrency climbs past 150. The scheduler answers one question every step, “does one more speculative token still pay for itself at this batch size”, and the answer moves on its own as the room fills up.

That is a throttle, and it is the good kind. It has the shape of every other scheduling lever in a serving stack, and it is better than most for one reason: it costs the user nothing. A priority tier slows somebody down so somebody else can be fast. DSpark’s scheduler withdraws a bet that was going to lose.

Truncating a block based on confidence is also the kind of thing that quietly breaks the losslessness guarantee, since an admission decision that peeks at a token it has not committed to leaks information into the sampling. The production version sizes the truncation from confidence measured two steps earlier, building a causal barrier between the decision and the tokens it could have cheated on. Even the load-dependent version reproduces the target distribution exactly.

Why any of this is our problem

We process a trillion tokens a week. We are self-funded, our budget is finite, and our pricing is deliberately generous, so how much work we get out of each GPU is what keeps all three true.

We cannot apply DeepSeek’s stack to GLM 5.2 by flipping a switch, but the techniques travel. A DSpark drafter trains against any frozen target model, and DeepSeek open-sourced DeepSpec, the repository that trains one; the paper does it for Qwen3 and Gemma4. Compressed attention is spreading just as fast, across Kimi, Qwen, MiMo and Ring. The models we already serve will arrive with this shape built in, and we wanted the operational answers before they do.

What we served

DeepSeek’s own MIT-licensed weights (1.6T total parameters, 49B activated), unmodified, plus the DSpark draft module. A 1M-token context window, three reasoning modes defaulting to think-high, text only. Access was seat-gated through the Labs page, with seats rotating so more people got a turn.

How it went

The speed held for the whole window. Our published metrics for the model put TTFT p50 at 1.37s at its best and 1.44s at close, against a 1.25s target, with throughput p50 at 181 tok/s per user against a 40 tok/s target. One tester ran a long agentic task through it, 153 tool calls end to end, and it finished in 12 minutes against roughly 20 to 25 for the same shape of task on our Kimi.

Deepseek followed instructions, didn’t loop or hallucinate at all. Isn’t nearly as verbose as GLM 5.2. Overall, extremely happy.

Fast. Not too deliberative. Seems to be making good decisions in the codebase I’m working in.

Then the failures arrived, and every one of them lives in the checkpoint.

Tool calls leaking as prose. DeepSeek V4 emits tool calls in its own DSML format, and testers repeatedly got raw DSML in the response body or buried inside the reasoning stream instead of a structured tool call:

The provider returned a tool call in the wrong channel and format: raw DSML inside reasoning_content instead of structured tool_calls. That violates the API contract the harness expects.

Language bleed. Chinese appearing in output, well outside the reasoning:

Hoping I’m not the only one seeing DeepSeek constantly trying to talk to you in Chinese, and not just in its reasoning either.

Precision decaying deep in context. At roughly 200k tokens, one tester watched it write SameSite: '/lax' where sameSite: 'lax' was meant, then catch its own mistake in the next breath. Another run produced untipdated for untyped. Others hit file edits failing often enough to eat the speed advantage:

feels like the speed of the model has been undermined by how many mistakes it’s made / its lack of capability, I’m pretty sure another model would have finished this by now

A malformed tool-call boundary, a language drift, edit precision degrading with context length: these are the signatures of a preview build. We were serving DeepSeek’s released weights with a distribution-preserving accelerator, so the checkpoint is the only place they could have come from.

The tool-call one is half ours to fix. DeepSeek’s format expects the reasoning block to close before a tool block opens, the model sometimes forgets, and the parser then swallows the entire tool call as reasoning so the harness never sees a structured call. vLLM merged a recovery for exactly this on July 4, treating the tool-start marker as an implicit end of reasoning. The equivalent fix in the engine we run is on our list, and we intend to send it upstream rather than carry it privately, since it will matter just as much for whatever DeepSeek ships next.

What we took from it

• DSpark lets us serve more users at once while keeping each session fast enough. The frontier moved, on live traffic.
• The DeepSeek V4 architecture is now mature enough to serve at scale. It runs, it holds a tail, and the serving-side unknowns we went in with came back answered.
• The model itself is solid. On its good runs it followed instructions, stayed on task, and did not pad.

V4 Pro stays out of the lineup, and the timing is most of the reason. Every problem testers hit is model-side, and DeepSeek has confirmed that a better version lands this month. Working around a preview checkpoint means spending the effort on weights that are about to be replaced. We would rather reach the real release already knowing how to serve it, which is what these two days bought.

What we ship

The compressed-attention idea is the one to watch, because it changes the arithmetic that has capped us all year. Attention that costs token × rep(token) instead of token × token means long sessions stop being expensive customers, and long sessions are what agentic coding is made of. DSpark sits on top and hands back the other half: it fills the idle time between tokens with work guaranteed to produce the model’s own output, and it knows to stop betting when the machine gets busy.

Given the choice between deciding which of you gets to be fast and making the machine serve more of you at once, we will pull the second lever every time. This is us learning how.

Thanks to everyone who spent two days throwing real work at a preview model and telling us bluntly where it broke. The reports in that thread were sharper than anything we would have found on our own, and they are the reason we can say with confidence which of these problems are ours to fix and which are not.

References

• DeepSeek-V4 technical report. Compressed Sparse Attention and Heavily Compressed Attention (§2.3), and the FLOPs and KV cache curves reproduced above.
• DSpark: load-aware speculative decoding. The confidence head, the hardware-aware scheduler, and the production frontier measurements.
• DeepSpec. DeepSeek’s open-source trainer for DSpark drafters, against any frozen target model.
• DeepSeek-V4-Pro-DSpark weights. MIT licensed, 1.6T total parameters, 49B activated. What we served, unmodified.
• MiMo-V2.5 inference. Xiaomi’s account of the memory-bound decode regime, quoted above.
• Prefill-as-a-Service. Qin et al., April 2026. KV cache reductions across the current hybrid-attention generation, and prefill shipped between datacenters.
• Our published metrics for the run. TTFT and throughput for the full two-day window.
• The Claude Code experience, self-hosted and GLM-5 vs Kimi-K2.5: long-context serving at scale. The long-context serving math this builds on.
• GLM-5.2 NVFP4: fast, cheap, and not worth serving. Why a checkpoint that degrades its own output does not ship, and why lossless acceleration is a different question.