Kimi K3, explained: how forgetting made the biggest open model possible
[
Kimi K3, explained: how forgetting made the biggest open model possible
In November 2025, the open frontier was a one-trillion-parameter model.
Kimi K2 Thinking could pick up a real GitHub issue, inspect the repository, patch the code, run the tests, and keep working through 200–300 tool calls without a person stepping in.
Eight months later, Moonshot released Kimi K3, a 2.8-trillion-parameter behemoth.
K3 is aimed at longer engineering jobs like:
- navigating large, messy repositories
- optimizing GPU kernels
- completing CAD (Computer-aided design) tasks
- and contributing to a chip-design demonstration.
The constraint behind all of those jobs is memory:
Kimi K3 inference path
The bill: memory that grows with everything at once
A model keeps a small bundle of working numbers for every token, at every layer and attention head. Engineers call this the KV cache. It saves the model from recomputing the past, but the cache grows with every token and must be read again to produce the next one. At frontier scale, that leaves less room for the model, fits fewer users on each GPU, and slows output as the conversation gets longer.
The memory bill: every word pays at every layer, for every head — and the total grows until it hits the chip
The cache bill affects every large model. Limited hardware access helps explain why Chinese labs had more reason to reduce it using software-based optimizations.
Memory efficiency was not optional
US labs have generally had more room to answer this problem by scaling hardware; Chinese labs, constrained by export controls on advanced chips, have had to squeeze more out of the memory already available.
DeepSeek's Multi-head Latent Attention, or MLA, was one answer. This approach compresses each token's stored key and value numbers into a much smaller representation to reduce memory usage. DeepSeek introduced it in V2, carried it into V3, and built R1 on the V3 architecture. It made every cache entry smaller, but every new token still added another entry.
MLA reduced the size of every stored entry, but it did not stop the number of entries from growing. K3 attacks both parts of the problem: Gated MLA preserves exact recall in one layer out of four, while Kimi Delta Attention gives the other three a fixed-size state that can be revised instead of extended. The four approaches below show how Moonshot arrived at that combination.
The toolbox: four ways to fight the memory bill
The four approaches are easier to compare if the task never changes. So we'll run one small meeting update example through all of them.
- Starting notes: The Acme planning meeting is Tuesday, in room 4B, with six people attending.
- New update: The meeting has moved to Thursday.
- What the model must predict: “The meeting is now on ___.”
The answer is always Thursday. Each tool gets there by carrying the earlier information forward in a different way.
What differs is what remains available when the model has to fill that blank. Share keeps fewer copies. Shrink makes each entry smaller. Forget keeps only recent entries. Replace trades the notebook for a fixed whiteboard that is edited in place.
The toolbox: share the notes (GQA), shrink them (MLA), forget the old ones (sliding window), or replace them entirely (linear attention)
1. Share the notes: GQA
A model has many attention heads looking for different things. One might care about dates, another names, another what changed. With ordinary multi-head attention, each head keeps its own copy of the keys and values. Grouped-Query Attention, or GQA, lets a group of heads share one.
In the meeting example, four heads look for the day, room, attendees, and latest change. They all read one copy of the notes and the model still predicts “Thursday.” GQA removes the duplicate copies, not the timeline. It is now the industry default, used by Llama, Qwen, Gemma, and Mistral.
GQA: four attention heads ask different questions of one shared copy of the meeting notes
Sharing removes duplicate copies, but the shared history still grows. MLA takes aim at the size of each entry.
2. Shrink the notes: MLA
Multi-head Latent Attention, or MLA, keeps the notebook but makes every entry much smaller. Instead of storing the full keys and values for each token, it stores a compressed latent representation and reconstructs what attention needs later.
The same Acme notes become a row of compact latent cards. The model can still recover “Thursday,” but every new token adds another card. DeepSeek's version shrinks roughly 32,800 numbers to 576, about 57× smaller. That cut its cache by 93.3% and pushed throughput 5.76× beyond its previous model.
MLA: each meeting note becomes a compact latent card, but the line of cards keeps growing
MLA makes each page cheaper without shortening the notebook. A sliding window caps the notebook itself.
3. Forget the old notes: sliding window
A sliding window puts a hard limit on the notebook. Keep the newest few thousand tokens and drop anything older when a new one arrives.
The desk tray has room for four meeting notes: client, original day, room, and attendees. When “moved to Thursday” arrives, the oldest card, “client: Acme,” slides out. The model can still predict “Thursday,” but that layer can no longer retrieve the discarded detail.
Mistral used sliding-window attention, and Gemma mixes five local layers with one full-attention layer.
The memory stops growing, but exact recall beyond the window needs another path.
Sliding-window attention: the newest meeting note enters as the oldest one leaves
A sliding window keeps memory flat by dropping old pages. Linear attention goes further: it stops keeping a page for every token.
4. Replace the notes: linear attention
Linear attention removes the per-token notebook. Each head keeps one fixed state, often pictured as a small grid, and updates that state as it reads.
The meeting example becomes a whiteboard with rows for client, day, room, and attendees. When “moved to Thursday” arrives, Thursday replaces Tuesday in the day row. The model predicts from the current state without preserving the exact sentences that produced it. The board does not grow.
This research line runs from Mamba through DeltaNet and Gated DeltaNet.
The memory stays fixed no matter how long the conversation gets, but matching full attention on exact recall remained difficult.
Linear attention: one meeting whiteboard is updated in place instead of adding another page
Sharing and shrinking preserve a growing history. Sliding windows and linear attention stop the growth by giving something up. Moonshot's next step was to test whether a fixed state could forget selectively enough to remain competitive.
Kimi Delta Attention: making the radical option work
Moonshot tested this idea in Kimi Linear, a 48-billion-parameter model released in October 2025 with public weights and code. Its attention mechanism is Kimi Delta Attention, or KDA.
KDA is easiest to understand as a recurrent state machine that runs once for every token. It carries one fixed-size memory state, S, forward. The key and value edit that state, the query reads from it, and the updated state becomes the starting point for the next token. The contents change. The size does not.
That fixed state is cheap, but it is less reliable when the model needs an exact sentence from far back. K3 keeps Gated MLA as the complementary recall path: three layers update the compact state, then one MLA layer can consult a compressed token-by-token history.
One KDA token step: k and v update a fixed-size recurrent state, q reads from it, and the state continues to the next token. The animation opens one KDA step. Token t becomes q, k, and v; k and v update the previous state S(t−1), q reads the updated state S(t), and the same-size state continues to token t+1. The notation matters less here than the loop. The state-update equation below will unpack what each term is doing.
Before opening the KDA mechanism, we need to know whether fixed-size memory actually works or merely costs less. Moonshot tested that directly. It trained two 48-billion-parameter models on the same data, at the same size, with the same recipe. The attention system was the only difference.
That turns the animation into a control experiment, not just an architecture sketch. The full-attention twin keeps adding pages. The KDA hybrid keeps most layers on a fixed-size whiteboard and gives one layer in four a compressed full-recall notebook. When the article later gets to 75% less memory, faster generation, and stronger 128K results, those numbers come from this head-to-head test.
The same conversation, twice: the notebook's memory climbs with every word; the whiteboard's never moves
Change one: erase, then write. Simply adding to a fixed-size state makes old and new information interfere. KDA's delta rule edits the state instead. For each token, it fades the existing memory, removes the old association for the current key, and writes the new value.
We are still inside the Acme example. For this close-up, the whiteboard holds three project facts:
- client → Acme
- budget → $80k
- and meeting day → Tuesday.
The next token says, “The meeting moved to Thursday.” KDA does not add a fourth row. It edits the meeting-day row in place: Tuesday is removed and Thursday is written into the same slot. The board ends with three rows, just as it began.
This is the core state update in KDA: the equation that decides what its fixed memory keeps, erases, and rewrites for every token.
The KDA update equation in motion: fade the state, erase meeting day → Tuesday, then write meeting day → Thursday
Remember α and β from the update equation? β controls how strongly the current key-value pair edits one association. α controls how much of the old state survives. The first change fixed how KDA edits memory. The second makes the forgetting control much less blunt.
Change two: 128 forgetting dials instead of one. Earlier gated delta models used one α value for an entire attention head, so every channel faded at the same rate. KDA uses 128 learned α values, one for each channel in the 128-wide state. The 128×128 whiteboard does not get larger. It simply spends its fixed memory more selectively: names and intent can persist while filler and pauses disappear.
The decay pattern also carries timing information. A signal that has been fading for longer looks different from a fresh one, which lets K3's MLA layers work without separate positional encoding.
Selective forgetting: one speed loses everything; 128 dials keep the essentials
The design call: keep an exact-recall path. Fixed-size memory remains less reliable when the model must retrieve a precise detail from far back. Kimi Linear therefore interleaves three constant-memory KDA layers with one DeepSeek-style MLA layer. Moonshot tested other ratios; the all-MLA version performed worst.
K3 keeps that 3:1 rhythm, upgrading the full-recall slot to Gated MLA. Three quarters of its attention layers use KDA, so most of the stack works from a fixed-size state instead of adding more notes for every token.
Every fourth layer retains MLA’s compressed record of the sequence, giving the model regular access to details that KDA may have blurred. The result keeps most of KDA’s memory savings without giving up a path for precise information recall.
One token through a K3 block: read KDA state, cross both residuals, route through 16 of 896 experts, repeat toward final logits
That hybrid only matters if fixed-size memory can keep up with full attention. Moonshot tested that on Kimi Linear before bringing it anywhere near the flagship.
What the swap bought
Everything here is measured on Kimi Linear against its identical full-attention twin (paper, model card, repo). These are the testbed's numbers, not K3's.
- Up to 75% less memory for the notes. Three layers in four store nothing per word, and the fourth stores 57×-compressed entries.
- Up to 6× more output at a million tokens. Most of that gain comes from larger batches: freed memory lets one chip serve several times more people at once. The paper's per-token latency figures put a single reply closer to twice as fast.
- 3.98× faster at 128K context, and it scored 84.3 on RULER, the standard long-memory test. The full-attention twin scored lower.
- No quality tax measured, on short tasks, long tasks, or reinforcement learning.
The outcomes, vs an identical full-attention twin: −75% memory, up to 6× output, faster and better at 128K, no quality tax
The 48B testbed showed that the hybrid could reduce memory without a measured quality loss. K3 carried the same design into a model 58 times larger.
Then Moonshot bet the flagship on it
The path to K3 had three important milestones: K2 at one trillion parameters in July 2025, a 48-billion-parameter KDA testbed that October, and K3 at 2.8 trillion parameters in July 2026.
The testbed showed that fixed-size memory could work before Moonshot carried the design into a model 58 times larger. K3 activates 104 billion parameters for each token, but the full model still has to support a million-token context window. If its notes kept growing at every layer, serving that window at open-model prices would be much harder.
Prove it small, then bet the company: K2 → the 48B testbed → K3 at 58× the size
Solving the cache was only the first part.
Inside K3: the other changes, each with an ancestor
KDA addresses how context memory grows, but it is only one part of K3's scaling plan. The other changes preserve information across layers, route work through 896 experts without overloading a few or saturating the communication links, and make a 2.8-trillion-parameter model practical to optimize and store. Each solves a different bottleneck created by scale.
The attention stack keeps Kimi Linear's 3:1 rhythm: three KDA layers, then one full-recall layer. K3 tightens both sides of that bargain. It stops KDA's memory from fading below a learned floor, which turns the update into a dense matrix operation GPUs handle well. In the fourth layer, Gated MLA adds a channel-wise output gate to the compressed full-recall path, controlling how much retrieved history enters the residual stream. Most layers maintain a compact working state; every fourth layer keeps a path back to exact wording.
Attention Residuals keeps earlier work accessible. A standard transformer pours every layer into one running stream. Many layers later, a useful syntax cue, fact, or plan survives only inside that mixture. K3 lets a later layer look back at earlier outputs and learn how much to take from each. The 0.6, 0.1, and 0.3 weights in the animation make that choice visible.
Attention Residuals, from the report's §2.2: instead of reading one blended pile, each layer queries earlier layers with learned weights
Stable LatentMoE makes 896 experts usable without running all 896. Each token wakes 16 routed experts plus two shared generalists, so only 104 billion of 2.8 trillion parameters are active. The latent projection keeps the messages sent between GPUs compact; otherwise, communication could eat much of the savings from sparse activation.
Quantile Balancing keeps the router from piling work onto the same experts. Rather than adding a hand-tuned penalty, K3 calculates a cutoff for each expert from its own routing scores. That gives the router a moving reference point for which experts are overloaded or underused, without adding another balancing knob. The experts do not get smarter; the work stops piling up on the same few.
Quantile Balancing, rebuilt from the report's Figure 5: loads of 4·3·1·0 — one expert starving — corrected to an exactly even 2·2·2·2
Per-Head Muon tunes each attention head separately. Muon is the optimizer: the rule that turns training errors into updates to the model’s weights. Standard Muon updates all the attention heads as one combined block, allowing a few heads with larger signals to influence the rest. K3 separates the heads and updates each one individually. One unusually strong head no longer sets the pace for everyone else. It is the same pattern seen elsewhere in K3: replace one coarse control with several smaller ones.
Four-bit weights cut what it costs to store and serve the model. Most models get compressed after training is finished, and quality slips when they do. K3 spent its entire post-training phase, learning with the 4-bit limit already in place, so it adapted to the constraint instead of being squeezed into it afterward. The compression targets the experts, which is where nearly all the weight sits. That is the difference between a 2.8-trillion-parameter model you can actually move between machines and one you cannot.
Almost nothing in K3 is a first draft: every component refines a proven precursor
Taken together, the pattern is local control. KDA learns decay per channel. Quantile Balancing corrects load per expert. Muon updates attention per head. Rather than trusting one setting to govern a large block, K3 gives smaller parts room to adjust independently.
A fixed-state attention system, 896 experts, and per-head controls only matter if training can exercise them together. K3's training recipe was built around that coordination problem.
How you even train this thing
The training recipe has three phases.
The first phase builds the base model. K3 trains on web text, code, mathematics, knowledge, and vision data. Images, video, and text pass through the same backbone. This is where it learns the broad language, reasoning, coding, and visual skills that later phases shape into agent behavior.
The second phase is real work. Reinforcement learning on coding, web agents, research, and professional knowledge tasks. A single episode can run hundreds or thousands of tool calls, so the model practices the loop the job needs: act, check, adapt. It also trains at several effort levels, which is why the released model can think briefly or think hard on request.
The final phase combines the specialist policies. Moonshot distills separate coding, agent, and reasoning behavior into one released model. The combined model learns from the decisions those specialist policies make on their strongest tasks. The result is one checkpoint that can move between coding, tool use, and reasoning instead of requiring a different model for each.
Three phases of schooling: pre-training on curated domains, RL apprenticeships on real long jobs, then many tutors distilled into one student
Moonshot released much of the machinery needed to make K3 work in practice:
- FlashKDA. Custom GPU kernels that make KDA's memory update fast on real hardware.
- MoonEP. An expert-communication system that distributes work across 896 experts during training.
- AgentENV. Resumable sandboxes from the kvcache-ai team, so long agent runs can recover after a crash instead of starting over.
The model, kernels, communication layer, and training environment have to work as one system; the architecture alone is not enough.
The infrastructure and the model were built for the same workload: long jobs that act, inspect the result, and continue. Those jobs make K3's memory design—and the price of reusing context—most visible.
What K3 is good at
Moonshot evaluates K3 on long jobs: day-long software engineering, web agents and browsing, professional knowledge work, and vision-in-the-loop coding where the model writes code, looks at the rendered result, and fixes it.
These jobs keep returning to the same codebase, documents, and tool definitions. That makes the cost of reusing context more useful than the headline context-window size.
Where K3 lands, per Moonshot's own suite: behind the two strongest closed models, ahead of everything else — built for the long stuff
The price advantage starts after the first request. Kimi automatically attempts to cache repeated initial context, with no cache ID or extra parameter. Put the system prompt, tool definitions, long documents, or codebase first. Keep that prefix unchanged, then append the task that changes. A miss costs $3 per million input tokens; a hit costs $0.30; output costs $15. A fully cached million-token working set is thirty cents to reuse.
That price does not come from KDA alone. KDA with prefill caching reduces the model's memory burden. Mooncake's serving architecture finds and reuses the stable prefix. An agent can return to the same repository or research corpus without paying to reread it from scratch on every turn. Moonshot reports cache hits above 90% for coding workloads. Your result depends on keeping the beginning of the prompt unchanged.
Moonshot also published two K3-built demonstrations. minitriton is a compiler, down to its own language front end, benchmarks, and documentation. nano-kpu is register-level hardware for an inference chip. Both require thousands of small corrections in sequence, each depending on earlier decisions.
In Moonshot's reported results, K3 ranks behind Claude Fable 5 and GPT-5.6 Sol, and ahead of every open model tested. The benchmarks show where K3 lands today, but the architecture is more useful as a way to read what comes next.
What survives the next release
Export controls raised the value of efficiency for Chinese labs. DeepSeek showed how much memory compression could change training and serving economics. Moonshot carried that pressure into K3's architecture and put its flagship on the result. The model names will change. The memory constraint will not.
Four ways to fight the bill: share, shrink, forget, replace. Every model you use has picked some blend of them. When a lab announces an enormous context window at a suspiciously low price, ask three questions. What got compressed? What gets thrown away? What happens to exact recall from half a million words back?
Those tradeoffs are the useful part of a model card: they show what the context-window number leaves out.
K3's answer is selective memory: preserve exact history in some layers and let the rest update a fixed state. That is how forgetting made a model this large practical.
Attention Residuals changes how information survives across depth, and it deserves a full explanation of its own. That is next.
For now, most of K3's stack is public. You can inspect the model, the Kimi Linear testbed, and the infrastructure that makes them run.
Go look at it yourself
Nearly all of this is downloadable today.
The model. The weights are on Hugging Face. The technical report contains the architecture details and the figures rebuilt in the animations above. If you'd rather call it than host it, there's the API quickstart. To serve it, vLLM and SGLang both published day-one recipes. Both currently run from dedicated Docker images and a branch; the upstream merges are still open.
Note: This Model is not MIT or Apache 2.0 licensed. See Kimi K3 License for more info
The engine room. FlashKDA for the attention kernels, with reference implementations in the flash-linear-attention library. MoonEP for expert-parallel training, and AgentENV for the agent sandboxes. Also PerceptionBench, the vision benchmark they built to test it, and Kimi Code if you want the terminal agent.
Start here instead. Kimi Linear, the 48B testbed where KDA was proven, with its own paper and weights. It's 48B total but only about 3B active per token, so it fits on hardware K3 never will. Honestly the better starting point if you want to run something yourself.
The ancestors, in the order I'd read them: Mamba, then DeltaNet, then Gated DeltaNet, and DeepSeek's MLA for the other branch.
Sources: Kimi K3 weights · Kimi K3 technical report · Kimi K3 tech blog · Kimi K3 developer docs · The Batch on K3's architecture · VentureBeat on K3 · Kimi Linear paper · Kimi-Linear-48B model card · MoonshotAI/Kimi-Linear code · DeepSeek-V2 / MLA · SCMP on DeepSeek's costs · Brookings on export controls · RAND on DeepSeek's lesson · vLLM / PagedAttention · GQA guide (Raschka) · Sliding-window attention in Gemma · Gated DeltaNet · DeltaNet, NeurIPS 2024 · Mamba