Kimi K3 from day zero: the largest model we operate, without a partner deal

On July 27, Moonshot published the weights for Kimi K3, and before that day ended K3 was live on umans.ai as a Labs pre-release. Six days later it graduated to general availability. This post is what happened in between: the largest model we have ever operated, five serving topologies, two engines, and a week of measurements that told us clearly which one to buy.

First, some context on how these launches usually work, because our day-zero was not the usual kind. Frontier labs now ship big models with an inference partner program: selected providers — Fireworks, Baseten, the names you would expect — get aligned ahead of the release and carry day-one traffic under a revenue-sharing arrangement. Moonshot’s own launch notes say they are working closely with inference partners and open-source maintainers for exactly that reason. We are not in that program. Nobody briefed us ahead of the release, and no revenue flows between us and Moonshot. What we had on July 27 was precisely what everyone else on the planet had: a public download. Day-zero serving came from open weights plus an inference stack that was already warm — and, credit where due, from SGLang and vLLM shipping K3 support the same day. None of this is a complaint about partner programs; they are how labs de-risk a launch. It is simply not what day-zero serving requires. Open weights are.

The largest model we operate

K3 is the first open 3T-class model: 2.8 trillion total parameters, 104B active per token, a sparse mixture-of-experts built on two architectural updates — Kimi Delta Attention (KDA) and Attention Residuals — with native vision and a 1M-token context window. The checkpoint is 96 safetensors shards, roughly 1.56 TB on disk.

That number drives everything below. An 8×B200 node carries 1,536 GB of HBM, so the weights alone do not fit in one — before a single token of KV cache exists. The smallest machines that can serve K3 are two B200 nodes, or one B300 node (8 × 288 GB = 2,304 GB), and the smallest useful production shapes are bigger than that. At roughly 1.75x DeepSeek V4 Pro’s total parameters and nearly 4x GLM-5.2’s, K3 is not just the largest model we operate; at $3.00 /$15.00 / $0.30 per million tokens (input / output / cache read) it is also the costliest per GPU-hour to keep honest.

The architecture softens one side of the problem. K3 is a hybrid linear-attention model: most layers are KDA — a delta-rule linear attention whose per-session state is fixed in size, like Mamba’s — interleaved with occasional full MLA attention layers whose KV still grows with the conversation. This is the compressed-attention generation we wrote about in the DSpark post, and K3 is its largest member. The serving consequence is that residency stops being only about KV tokens and becomes about state slots: each live session holds its small FP8 MLA KV plus one constant-size BF16 KDA state, whether the conversation is ten tokens or nine hundred thousand. Long sessions stop being expensive customers in exactly the way we hoped — but the weights got so big that the constraint moved there instead.

We process a trillion tokens a week, we are self-funded, and our pricing is deliberately generous. A 2.8T model served on the wrong topology is a bonfire. So we did what we always do: made it work on the smallest setup possible, then ran the candidate shapes against each other on real workloads.

First: make it work

The smallest setup that serves K3 is two 8×B200 nodes, and the fastest way to stand it up was the shape we already operate for other models: vLLM, TP8 × DP2 × EP16 — one engine replica per node, experts spread across all sixteen GPUs, sessions pinned to the replica that owns them, and a Mooncake-style L2 pool (640 GiB of DRAM across the pair, over EFA/RDMA) so a session evicted from one replica’s VRAM can be rehydrated from host memory instead of recomputed.

Within a day of the weights landing, that pair was passing tool-calling gates. By the end of the qualification window it had:

• the full 1,048,576-token context, verified end to end: a 1,047,888-token prompt reached first token in 150.7s cold, and its warm follow-up — same session, L1 hit on 1,046,016 of 1,047,888 tokens — in 1.96s;
• vision intact;
• Synbad 44/44: OpenAI chat and Anthropic Messages, streaming and not, eleven tool cases each;
• DSpark active, with 47% of drafted tokens accepted on live traffic.

It worked. It also told us immediately that working is not paying: the pair’s safe measured admission is two concurrent agentic trajectories, globally. Repeated trials at four and eight wedged at 95–98% KV occupancy and stopped making progress. Inside that envelope it is genuinely good — the final admission-policy run held 50–64 tok/s per user (decode p10), 1.22s TTFT p90, and 89.7% prefix-cache reuse on replayed agentic traffic — but two sessions per pair is a lab result, not a product. Time to iterate.

Then: three setups at once

We ran the next phase as three concurrent experiments. All three trace back to the same idea, which has the best paper trail in serving: prefill and decode want different things. Prefill is compute-bound, decode is memory-bandwidth-bound, and colocating them makes each worse at the other’s job — DistServe (OSDI ‘24) named the goodput math, Splitwise (ISCA ‘24) showed phase-specific hardware assignment, and Moonshot’s own Mooncake — the platform that serves Kimi itself — is the most famous disaggregated deployment in production. There is a pleasing symmetry in it: we ended up serving Moonshot’s model on the architecture their own serving paper describes, assembled from SGLang and vLLM parts. The KV handoff rides the fastest fabric available — NVLink inside a node, RDMA across nodes; on our AWS pairs that is NIXL or Mooncake over EFA, sixteen interfaces per pair.

Setup one — 2×8 B300, disaggregated prefill/decode (SGLang). TP8 prefill on one node, TP8+DCP8 decode on the other, KV transferred over NIXL/EFA. The prefill node carries a HiCache DRAM tier at ratio 12 — about 3 TB of host cache, 24.8M logical tokens — so agentic sessions that go quiet between turns rehydrate from DRAM rather than reprefill. Prefill chunks are 65,536 tokens, so a cold 100K prompt is two chunks instead of seven. This is the setup that went to production: qualified July 29 with zero restarts, admitted as the K3 pool member, and its first 33 real production requests completed without a failure. Synbad passed 42/44 — both misses the same semantic case (an optional dirPath argument the model supplied and the strict test expected omitted), in both Anthropic modes.

Setup two — 1×8 B300, unified (SGLang). The simplest thing that fits: the whole model in one box, TP8+DCP8, FP8 KV, HiCache at ratio 8, no disaggregation. Its qualified admission point is 16 simultaneous cold 8K prefills while holding a 32 tok/s decode floor (37.7 measured); 64 is the engine’s state-slot ceiling, not a cold-prefill concurrency. One detail worth stealing: enabling mixed prefill/decode chunking cut the worst observed inter-token stall from ~22 seconds to 1.26 seconds. If you run one big box, turn that on.

Setup three — 2×8 B200, two ways to split 2.8T across sixteen GPUs. Same hardware as the make-it-work pair, two different philosophies:

• SGLang TP16/DCP16 — one engine spanning both nodes, every session’s KV sharded across all sixteen GPUs. This is the capacity monster: 22.7M logical resident tokens with no speculation and the radix cache on — roughly 150 live 150k-token sessions, or about 21 full-1M sessions, on a pair. It is also, at batch size one, the fastest single stream we measured on B200 (79.5 tok/s output, 83 decode p10): one request gets sixteen GPUs of memory bandwidth. (A predecessor topology, DP2/DCP8 with HiCache, capped single requests at 65k tokens and never finished OCRBench; we keep it as evidence, not as a candidate.)
• vLLM TP8×DP2×EP16 — the make-it-work pair, measured as a product: the fastest per-session experience on agentic traffic inside its tight admission envelope, with Mooncake L2 keeping follow-ups warm across replicas.

Here is all of it on one chart. Every point is the same deliberately harsh workload — 8,192 input / 1,000 output tokens per request, cache flushed before every round — so the differences are the setups, not the dice:

Three things to read off it. One: the B200 pair saturates around 400–430 output tok/s no matter how we split the model — above BS16 it stops getting faster and starts queueing (hollow points). Two: a single B300 node beats the whole B200 pair at its qualified point (552 vs 368 tok/s at BS16) and extends past 900 pushed. Three: disaggregation buys the top-right corner — the P/D pair reaches 1,163 output tok/s at BS64 while every user still decodes at 32 tok/s or better, which is exactly the DistServe thesis measured on our own traffic shape: split the phases and the per-user floor survives concurrency.

The per-GPU math is the part that pays for things. At matched batch size 16: 69 output tok/s per B300 GPU (unified) versus 23 per B200 GPU — a 3x gap. B300 wins on cost under any pricing where a B300 GPU-hour costs less than 3x a B200 one; the market rate is about 1.4x. It is not close.

Draft models: measured on both engines, rejected on both

K3 ships no draft tensors — no MTP, no EAGLE — so the only speculative path today is an external block-diffusion draft, RadixArk’s Kimi-K3-DSpark. We gave it a fair trial on both engines. The short version: the overhead exceeds the benefit at the concurrency we serve.

On SGLang we ran the comparison properly: same pair, same public agentic trace, same seed, fresh engine and empty cache on both arms, concurrency 8. Static DSpark against no-spec:

The mechanism is the same one DeepSeek’s DSpark paper warned about, and we measured it biting: K3 on TP16/DCP16 currently forces static verify-all — the engine verifies eight positions to keep 1.2 tokens, and the draft’s replicated state cuts residency by 5x. The load-aware compact verification that made DSpark worthwhile in DeepSeek’s production is not validated upstream for K3’s hybrid DCP path yet (the relevant SGLang PRs are open, not merged). We chased it properly anyway: ReplaySSM recovered some throughput but not the capacity, and the best static configuration we found (gamma 1, verify 2: 148.6 tok/s) still trailed no-spec by 14.5% with 5.5x less residency. No-spec plus radix is the measured recommendation.

On vLLM the draft behaved far better — 47% acceptance on live traffic is genuinely good — and it still was not worth it: carrying the draft costs about 30% of the pair’s resident capacity (3.29M → 2.32M logical tokens), and admission stayed C2 either way. Paying a third of your session capacity to speed up sessions you cannot admit is the wrong trade at our load.

Two honesty notes. First, this does not contradict Wafer’s K3 numbers, where the same draft bought ~2.2x single-stream on MI355X: at low concurrency, rejected drafts cost compute that would have idled anyway, and speculation wins — the paper says so, and their measurement agrees. The divergence is the regime, not the rig. Second, we expect this to flip: once confidence-scheduled compact verification lands for K3, we will re-run the whole comparison, because the DeepSeek result says the load-aware version is a different animal.

How the numbers compare

The best public K3 serving numbers we know of are Wafer’s (1,024 in / 400 out benchmark), and ours should be read next to them:

Read the caveats before the digits. Our per-stream numbers carry no speculative decoding; theirs are draft-assisted. Our workload fronts 8x the prefill volume with caches flushed between rounds; a cold 8K prefill taxes exactly the phase a short-benchmark number never feels. And we measured to decide admissions — every umans point holds its per-user floor at the stated concurrency — not to chase a peak. Don’t over-read either table. The agreement is the interesting part: both labs land B200 pairs in the same few hundred tok/s, both find B300 several times better per GPU, and both are in the same league on stream speed once you account for the draft and the workload.

What we took from it

• Prefer B300 when you can. It is the clear winner under every setup we measured — most cost-effective and best performance, about 3x per GPU at matched SLO. There is no B200 arrangement that closes that gap.
• The production shape is a disaggregated cell: 4×8 B300, prefill/decode split. Prefill and decode nodes in the ratio your traffic’s input:output mix dictates — and then as many cells as your availability SLOs require, because one cell is one failure domain. We qualified the 1P+1D half of that cell in production; going to four nodes is the same engineering with the ratio knob turned, plus the redundancy math.
• On B200: vLLM for interactive, SGLang for capacity. The vLLM pair gives each session the fastest ride — 50–64 tok/s p10 per user, ~90% cache reuse, sub-1.3s TTFT p90 — inside a small envelope. The SGLang TP16 pair’s envelope is ~10x larger (22.7M resident tokens). Fastest per session versus most sessions: both are real products, and we run each for what it is good at.
• Draft models are overhead today, on both engines. Not because the idea is wrong — because the load-aware version hasn’t landed for K3’s stack yet. We will re-measure when it does.

What we ship

Kimi K3 left the Labs pre-release and is generally available since August 1: no seat required, 1M context, native vision, max-effort reasoning, at $3.00 /$15.00 / $0.30 per million tokens. Production runs the B300 disaggregated pair, and its live TTFT, throughput, and uptime are public on our status page — the same page that carried the experiment while it ran.

Everything above is reproducible: the exact engine pins, patches, admission math, and raw evidence for all five topologies live in the Kimi K3 experiment index of our llm-gateway repository. The patches we carry are small and boring on purpose — admission accounting, allocator corrections, a fused-KDA kernel guard — and what isn’t already merged upstream is on its way (the draft-pool and allocator fixes we rode are in SGLang #32828, merged August 1). What’s next on our list: compact verification when upstream qualifies it, the blocked PP8 prefill lane, and a Mooncake-style L2 for the SGLang shapes.

Thanks to everyone who threw real work at the pre-release while it was still finding its footing, and to the SGLang, vLLM, and Moonshot open-source teams whose day-zero support is the actual reason day-zero serving is now something any focused team can do — partner program or not. We know, because we did.

References

• Kimi K3 technical blog and weights. 2.8T total / 104B active parameters, KDA + Attention Residuals, native vision, 1M context. What we served, unmodified.
• DistServe: disaggregating prefill and decoding for goodput-optimized serving. OSDI ‘24. The goodput math behind phase splitting.
• Splitwise: efficient generative LLM inference using phase splitting. ISCA ‘24. Phase-specific hardware assignment.
• Mooncake: a KVCache-centric disaggregated architecture for LLM serving. FAST ‘25. Moonshot’s own production platform for Kimi — and the L2-pool design our vLLM pair borrows.
• Is memory the moat? Running Kimi K3 at ~952 tok/s/node. Wafer’s MI355X/B200/B300 comparison — the public numbers we benchmarked ourselves against.
• RadixArk’s Kimi-K3-DSpark draft. The only speculative path for K3 today; measured and rejected on both engines above.
• DeepSeek V4 Pro DSpark: the model isn’t ready, the architecture is. Why load-aware speculative decoding is the version worth waiting for.
• The Kimi K3 experiment index. Every runtime, qualification record, and raw-evidence manifest behind this post.
• SGLang #32624 (fused KDA for six-head TP16) and #32828 (DCP/DSpark draft pool, verify workspace, allocator). The upstream fixes our B200 SGLang candidate pins.
• Our published metrics for the model. Live TTFT, throughput, and uptime, from pre-release through GA.