---
title: "Kimi K3: Two point eight trillion parameters, open weights, and forty-seven pages of how"
url: "https://chriscaruso.dev/notebooks/kimi-k3"
type: "Notebook (interactive explainer)"
site: "Caruso's Conjecture"
author: "Chris Caruso"
published: "2026-07-27"
source: "writing/notebooks/kimi-k3.md"
index: "https://chriscaruso.dev/llms.txt"
narration_minutes: 65
narration_cues: "https://chriscaruso.dev/media/narration/kimi-k3/cues"
narration_chapters: "https://chriscaruso.dev/media/narration/kimi-k3/chapters"
tags: ["Machine Intelligence", "Architecture", "Attention", "Benchmarks", "Interactive"]
---

# Kimi K3: Two point eight trillion parameters, open weights, and forty-seven pages of how

**Two point eight trillion parameters, open weights, and forty-seven pages of how.**

Kimi K3 went live on 16 July 2026 and became legible on 27 July, when Moonshot AI published the weights, the config, a new license and a 47 page technical report.[^1][^2][^3] It is the first open model at the three trillion parameter scale. It is also, as of this writing, the most completely documented frontier model that exists, and the second of those is by far the rarer achievement.

This notebook reads the whole thing end to end. What the model is, why each piece is shaped the way it is, how it was trained, what runs underneath it in production, what it scores, what it costs, where it breaks, and what Moonshot still has not said.

The thesis fits in one paragraph, and it is Moonshot's own. A transformer moves information along three axes, and K3 attacks all three at once. Along the **sequence**, Kimi Delta Attention replaces three quarters of the attention stack with a fixed size recurrent state. Along **depth**, Attention Residuals replace the uniform residual stream with a learned softmax over every layer below. Along **width**, a Stable Latent MoE routes 16 of 896 experts through a compressed latent space. Everything else in K3 is in service of one of those three, or of making all three survive contact with a datacenter.

## Twelve months

K3 did not arrive from nowhere. It is the terminal point of a year in which Moonshot published, in sequence, every component it is built from.

> **Interactive figure:** A timeline of Moonshot AI releases from Kimi K2 through the K3 weights.

The pattern is worth naming. Kimi Linear, in October 2025, was a 48 billion parameter research model whose entire purpose was to validate one attention mechanism at a scale nobody would deploy.[^4] Attention Residuals, in March 2026, did the same for one change to how layers talk to each other.[^5] Both papers ran their experiments, published their ablations, and stopped. Neither mentions K3, because neither knew about it.

The second paper is more useful than that summary suggests. Its architecture section describes the model it trained as identical to Kimi Linear, KDA interleaved with full attention at three to one, with AttnRes on the residual connections as the only modification.[^5] So the two mechanisms were not validated separately and combined for the first time in K3. They were already running together, at 48 billion parameters, four months before K3 was announced.

K3's contribution is not the combination. It is the combination at fifty eight times the size, plus a third axis the research models did not have.

## What it actually is

Here is the model, from the config file.

> **Interactive figure:** Kimi K2 against Kimi K3, field by field.

Some of these numbers are worth sitting with.

**The parameters are almost entirely experts.** Multiply out the config and the 896 routed experts across 92 MoE layers account for 2.723 of the 2.78 trillion parameters. That is 97.9 percent of the model.[^6] Attention is 33 billion. The embeddings are 2.3 billion. The dense layer at the bottom of the stack is under a billion. When you say "2.8 trillion parameter model" you are, to two significant figures, describing a bank of feed-forward experts of which 98.2 percent are asleep for any given token.

**The model got deeper, not wider.** K2 was 61 layers at hidden dimension 7168. K3 is 93 layers at hidden dimension 7168. The residual stream is exactly the same width it was a year ago. Every additional parameter went into depth, into expert count, or into the latent space inside the MoE. This matters more than it looks, because a wider residual stream is the easy way to scale and Moonshot declined to take it. They scaled the three axes instead.

**The head count went up while the attention cost went down.** 64 heads to 96, and yet attention is a smaller fraction of the model and a far smaller fraction of the inference bill, because 69 of the 93 layers no longer keep a KV cache at all.

**It fits on disk.** The full checkpoint is 1,453 GiB across 96 safetensors shards.[^2] A 2.8 trillion parameter model that downloads in under one and a half tebibytes is only possible because the experts are natively four bit, which is the first of several places where K3's architecture and K3's deployment turn out to be the same decision.

## The sequence axis

Attention is quadratic because every token attends to every token. Linear attention is not, because it does not keep the tokens. It keeps a fixed size state matrix and writes each token into it as it goes. Reading is a single matrix multiply against that state, no matter how long the sequence is.

The problem is what "writes into it" means. The crude version adds, and things you add never leave. Kimi Delta Attention is the fourth step in a four step argument about how to fix that.

> **Interactive figure:** Four update rules writing tokens into a fixed size state, stepped one token at a time.

The four rules, in order: **linear attention** adds every token to the state forever. The **delta rule** subtracts what the state already holds along the incoming key before adding, so writing a key twice overwrites rather than doubles. **Gated delta** multiplies the whole state by a scalar between zero and one at each step, so old material decays. **KDA** makes that gate a vector, one decay per feature channel, so the model can hold a variable name in some channels while flushing a loop counter from others.

That last step is the entire mechanism, and stated plainly it sounds too small to matter. It is a scalar becoming a vector. What it buys is the ability to forget selectively: the thing a fixed size memory needs most, and the thing a scalar gate cannot express.

### What changed between the paper and the model

K3's KDA is not the KDA in the Kimi Linear paper. Two things moved, and both moved for reasons that have nothing to do with modelling quality.

**The decay is now bounded from below.** Kimi Linear parameterised the per channel decay through a negative softplus, so the log decay ranged over all of the negatives and the decay factor could get arbitrarily close to zero. K3 replaces that with a lower bounded scaled sigmoid: the log decay is a sigmoid scaled by a floor of minus five, so the per step decay lives in a range from about 0.0067 up to 1.[^1]

The motivation is a kernel motivation. The efficient way to compute a gated delta rule over a tile of tokens is to divide out the cumulative decay, do dense matrix multiplies, and multiply it back. If the cumulative decay can approach zero then its reciprocal can approach infinity, and in BF16 that overflows. Bounding the decay at e to the minus five keeps the reciprocal under e to the eightieth over a sixteen token tile, which is inside BF16's dynamic range, which means the diagonal tiles can be computed with dense Tensor Core matmuls instead of the explicit position pair path Kimi Linear needed. A modelling hyperparameter was chosen so that a GPU instruction would be legal.

**The output gate went from low rank to full rank.** Kimi Linear used a low rank projection to produce the output gate. K3 uses a full rank one. So does the Gated MLA. This is a straightforward spend of parameters on expressiveness at a point where the parameters are cheap relative to the expert bank.

One more detail from the same section, small and telling: during training, K3 keeps the flash attention output in FP32 rather than casting down, because the standard BF16 accumulation introduced a rounding bias the model was learning around.

### Three KDA layers, then one full one

A fixed size state cannot hold everything, so K3 does not ask it to. 69 of the 93 layers are KDA. 24 are Gated MLA, full attention with a compressed key value cache and an added output gate. The pattern is three KDA layers then one MLA layer, repeating.

The config confirms the ratio and then adds a wrinkle nobody would have guessed. The full attention layers are 4, 8, 12, and so on up to 92, and then also 93. There is an extra Gated MLA layer bolted onto the very end, so layers 92 and 93 are both full attention, back to back. The report's reason is as blunt as it sounds: the final layer of the backbone should always perform global attention.[^1]

The MLA layers also drop positional encoding entirely. The config sets NoPE for MLA, which means the only thing in the model that knows where a token is, is KDA's decay. Position is carried by the recurrence, not by a rotation, which is why K3 could be extended to a million tokens without any of the rotary embedding rescaling tricks that long context extension normally requires.

### The bill for that

> **Interactive figure:** Cost of full attention against KDA as the context grows.

The crossover is the honest part. Linear attention is not free, it is constant, and a constant beats a linear cost only after the linear cost has grown enough. Below roughly five hundred tokens KDA is the more expensive choice. At a million tokens it is not close.

Three quarters of the layers keeping no KV cache is what makes a million token context a product rather than a demo, and it is what makes the cached input price on the pricing page possible. The architecture and the invoice are the same decision seen from two directions.

## The depth axis

The second axis is stranger, and it is the one I would nominate as K3's most interesting idea.

A standard transformer's residual stream is an unweighted running sum. Layer 40 reads the sum of everything the 39 layers below it wrote, each contributing with weight exactly one. Nobody chose that. It is what falls out of writing `x = x + f(x)` and it has been the default since 2015.

Attention Residuals ask what happens if the model gets to choose instead.

> **Interactive figure:** Uniform residual accumulation against a learned softmax over depth.

The mechanism is attention, run over depth instead of over sequence. Each layer emits a query. Every layer below it has emitted a key. Softmax over those gives a distribution over depth, and the layer reads a weighted mixture of the outputs below rather than their raw sum.

Two consequences follow immediately. The first is that the total weight is now exactly one at every depth, because softmax normalises, which means the residual stream's magnitude stops growing with depth. That is a stability result, and stability at 93 layers is not a nicety.

The second is the one nobody designed. Because it is a softmax, the model can put mass wherever it likes, and what it reliably does is put a persistent chunk of mass on the token embedding at the very bottom. A depth-wise attention sink, structurally identical to the sequence-wise attention sinks that show up on the first token in ordinary attention, discovered by the same mechanism for presumably the same reason. The report notes, without remarking on the resemblance, that the token embedding is always available as a source.

K3 uses the deployment variant, **Block AttnRes**, with a block size of 12.[^2] The layers are grouped into blocks, each block contributes one summed key and value, and a layer attends over blocks rather than over layers. At 93 layers that is 8 blocks, the last one partial, plus the token embedding: nine depth-wise sources. Memory goes from linear in depth to linear in block count, cheap enough to run in production. The implementation merges the inter-block and intra-block partial results with an online softmax, exactly the way flash attention merges tiles.

## The width axis

The third axis is the one the earlier research models did not have, and it is where nearly all the parameters live.

> **Interactive figure:** One mixture-of-experts layer, showing which experts light up for a token under K2's sparsity and under K3's.

K2 chose 8 of 384, a sparsity of 48. K3 chooses 16 of 896, a sparsity of 56. Both counts went up, the ratio went up with them, and the fraction of the layer doing work on any given token went down to 1.8 percent.

**Stable Latent MoE** is what makes that ratio survivable. In an ordinary MoE, every expert is a pair of matrices as tall as the residual stream, so 896 of them at hidden dimension 7168 would be ruinous. K3 puts a bottleneck in front of the bank. The layer projects the 7168 dimensional stream down into a 3584 dimensional latent space, runs the experts entirely inside that space, and projects the result back up. That halving is the "Latent" in the name, and it is what pays for the expert count: each expert's matrices are half as tall as they would otherwise be, so you can buy twice as many for the same money.

The "Stable" half is an RMSNorm at the boundary before the up-projection, doing what the name suggests, keeping the magnitude of what leaves the latent space from wandering as the expert count grows. Two shared experts also run on every token regardless of what the router decides, giving the layer a floor of always-on capacity underneath the sparse part.

Two more pieces of the width story deserve their own naming, and both are better than they sound.

**SiTU-GLU** replaces SwiGLU as the activation, and the appendix is worth reading for how carefully it is bounded. SwiGLU multiplies a gate branch by an up branch, and neither branch is bounded, so their product can run away. SiTU caps each branch with a scaled hyperbolic tangent, at 4 for the gate and 25 for the up branch. Since tanh and the sigmoid are each bounded by one, every output coordinate is bounded by the product of those two shape parameters: exactly 100.[^1]

Two properties make that cap nearly free. Near the origin a scaled tanh is the identity to first order, so SiTU-GLU agrees with SwiGLU wherever activations are small, and activations are almost always small. And because the cap is smooth rather than a hard clamp, gradients stay nonzero as a unit approaches the ceiling instead of dying against it. That is the whole trick: a ceiling that changes nothing until it is needed. Bounding an activation function is a training stability decision, and the pattern by now should be familiar. K3 is a model where a startling number of design choices are stability choices wearing an architecture costume.

**Quantile Balancing** is the more elegant of the two, and it is the clearest example in the report of Moonshot explaining a technique everyone already uses. Every large MoE has to stop its router funnelling traffic to a few favourite experts. The current standard is auxiliary-loss-free balancing: give each expert a bias, and after every step nudge that bias up or down by a fixed amount depending on whether the expert came in under or over its share.

The appendix derives where that rule comes from. Write down the real problem, assigning tokens to experts so total routing score is maximised while every expert receives exactly its share. Relax it to a linear program, take the dual, and the exact solution for each expert's bias turns out to be a quantile of the scores it is competing for. Both sides of the alternating solver are the same quantile, taken once along the token axis and once along the expert axis. Hence the name. The familiar nudge-by-a-fixed-amount rule is what you get by taking a sign-gradient step on that same dual objective. Quantile Balancing skips the step and jumps straight to the exact minimiser.

Two things follow. It has no learning rate to tune, because it is not doing gradient descent at all. And it equilibrates within a handful of steps even at nearly a thousand experts, which is precisely the regime where a fixed-step nudge is slowest. Computing an exact quantile over millions of tokens sharded across a cluster is impossible inside a training loop, so K3 estimates it from a per-expert histogram of 1,000 bins, costing one integer all-reduce per layer per step and landing within a few thousandths of the true value. Because counts add, the estimate is exactly invariant to how tokens were split across machines. And only the expert-side thresholds survive into the finished model, frozen as a bias, so serving is an ordinary top-16 selection with no quantile machinery anywhere near it.

The result is a router balanced by construction rather than by pressure, which turns out to matter enormously to the infrastructure, as we will see.

## Eyes

K3 is natively multimodal, not a language model with an adapter. The distinction is concrete: language and vision are optimised together from the first step of pretraining, rather than a vision encoder being trained separately and then grafted onto a finished language model. The encoder is **MoonViT-V2**: 401 million parameters, 27 layers, 12 attention heads, patch size 14, accepting images up to 3584 by 3584 pixels.[^1][^2] Between it and the language model sits a 2x2 pixel shuffle, which folds each two-by-two block of neighbouring patches into a single position with four times as many channels. Nothing is discarded; the image simply arrives as a quarter as many tokens. Without that fold, feeding in a 3584 pixel image would not be affordable at all.

The interesting part is how it was pretrained, or rather how it was not. Contrastive pretraining, of which SigLIP is the usual instance, teaches an encoder by pulling matching image and caption pairs together in a shared space and pushing mismatched pairs apart. It has been the standard first step for essentially every serious vision-language model of the last three years, and it is where almost everybody gets their encoder. Moonshot trained MoonViT-V2 from scratch with next token prediction and no contrastive stage at all.

The reason they give is stability. The SigLIP initialised run showed persistently higher gradient norms with periodic spikes, and the from-scratch run did not. And having trained both, they report the from-scratch encoder matches the SigLIP baseline on vision evaluations. The conclusion they draw is a negative result stated plainly: at this scale, contrastive pretraining is unnecessary as a multimodal initialisation. That is a load-bearing claim for anyone building the next one of these, and it is the kind of thing that usually never gets published because it makes a standard practice look optional.

## How it was trained

The pretraining section is where the report is simultaneously most detailed and most careful about what it will not say.

**The corpus is four text domains and a vision one.** Web text, code, mathematics and knowledge, each put through rule-based filtering, classifier quality scoring and deduplication, with the mixing rates between them decided by ablations on smaller models rather than by intuition. Knowledge and mathematics are additionally rephrased in varied styles and perspectives and then checked back against the source document for fidelity, a recipe carried over from K2.

The vision half is where the interesting bet is. Alongside ordinary captioned images and OCR, Moonshot says it substantially scaled up **programmatic multimodal data**: code paired with what that code renders, across SVG, 3D assets, web pages, games and CAD schematics. Object coordinates are supervised in both absolute pixels and normalised 0 to 1 form, so localisation survives a change of resolution. That is the only line in the report's data section that predicts a benchmark result, and it predicts the most lopsided one K3 has. The model is first on WebDev Arena, and its widest margin under blind expert judging is on 3D and shader work. It was fed a corpus built to teach precisely that mapping, from source text to rendered pixels.

**The optimizer is Per-Head Muon.** Muon orthogonalises gradient updates using Newton-Schulz iteration rather than treating weights as a flat vector of scalars. K3's variant applies that orthogonalisation to each attention head's block independently rather than to the whole projection matrix at once. And it runs, in the report's words, together with the weight clipping mechanism introduced in Kimi K2. So MuonClip survived into K3 without being named. It is now a clause in a sentence about something else.

**The learning rate schedule is cosine, and the justification is the interesting bit.** Warmup-stable-decay has become close to universal for large runs because it lets you branch a cooldown from any point. Moonshot ran a comparison and picked cosine, but the part worth stealing is the methodology: they performed an independent hyperparameter search for each schedule before comparing them, on the grounds that comparing two schedules at hyperparameters tuned for one of them is not a comparison. That is an obvious point that almost nobody honours.

**The context curriculum has four stages.** 8K to 64K during pretraining, then 256K to 1M during cooldown. No positional interpolation tricks are needed, because the MLA layers use no positional encoding and KDA's notion of position is its decay.

**The scaling law was retuned.** The report claims roughly a 2.5x improvement in overall scaling efficiency over K2. That single figure is what the whole architecture is ultimately justified by. Tokens per parameter was re-derived rather than inherited.

And then the absences, which are worth stating explicitly because everything around them is so thoroughly documented. **The number of pretraining tokens is not published.** Neither is the pretraining hardware, the cluster size, the wall clock time, or the cost. K2's report gave a token count. K3's gives a scaling law and a methodology and no absolute number anywhere.

Two crumbs do escape, both by accident, and both from sections about something else. The infrastructure chapter mentions keeping each million-token RL experiment "within a few hundred GPUs", a bound on the reinforcement learning budget if not the pretraining one. And the evaluation appendix twice mentions running GPU-dependent benchmarks on **H20** accelerators, the export-compliant part Nvidia sells into China, in place of the H100s those benchmarks officially specify. Neither is a disclosure. Both are the kind of thing you only say when you are not thinking about what you are saying, which is why they are more informative than the sections that are trying to inform you.

## Nine models, then one

Post-training is where K3 stops resembling its predecessors.

The standard recipe has two stages. Supervised fine-tuning teaches the raw pretrained model to behave like an assistant by imitating good examples. Reinforcement learning then improves it against a reward, letting it generate, scoring what it generated, and pushing it toward whatever scored well. K3 follows that recipe and then does something unusual at the end of it.

The reinforcement learning stage does not produce one model. It produces **nine**: three domains crossed with three reasoning effort levels. Each of the nine is specialised, and each is better at its own corner than a single model trained on everything could be at all nine corners at once. That is the familiar tradeoff, and normally it is where you stop and pick one.

Instead, **Multi-Teacher On-Policy Distillation** collapses all nine back into a single checkpoint. The student generates a response, the appropriate teacher scores it token by token, and the reward is the log ratio between what the teacher would have said and what the student did say, clipped so no single token can dominate. The word doing the work is *on-policy*. Ordinary distillation trains the student on transcripts the teacher wrote, which means the student is graded on text it would never have produced; here the student is graded on its own output, so it only ever learns about the distribution it actually occupies. Moonshot notes they also tried finer-grained top-k distillation objectives and saw no benefit. That is the kind of negative result that saves someone else a quarter.

### The mechanisms

**Reasoning effort is a trained budget, not a prompt.** Each problem gets an estimated token budget, and the reward penalises exceeding a multiple of it. Annealing that multiple downward during training is what produces the low and high effort variants out of the same run. So when you set `reasoning_effort` on the API you are selecting between behaviours that were separately optimised, not asking the model politely to be brief.

**Partial rollouts fix the straggler problem.** Long horizon agentic rollouts vary enormously in how long they take to finish, and a batch is only done when its slowest member is. K3's RL pauses generation once some fraction of the trajectories in a batch have completed, and resumes the rest in a later iteration. This is the difference between an RL step costing as much as its slowest trajectory and costing as much as its median one.

**The reward model has a verbosity budget.** The agentic reward model does not score trajectories in isolation; it runs a tournament of binary comparisons under a mandatory rubric, and enforces a length budget on top. The reason is that the reliable failure mode of optimising against a judge is that outputs get longer, because length correlates with thoroughness until suddenly it does not. Given that verbosity is one of the standing criticisms of K3 in practice, this is a mechanism that was built and then only partly worked.

**And the model is quantised the whole time.** Quantization-aware training in MXFP4 runs through all of SFT and all of RL, with expert weights in four bit and their input activations in MXFP8. Rollout and training share the same quantization scheme, so there is no train-inference mismatch left to correct at the end. The four bit weights are not a compression applied to a finished model. They are the weights the model was trained to have.

The scope is narrower than "the model is four bit", and the config is exact about it. The MXFP4 group applies to Linear layers with a group size of 32 and a uint8 scale, and the ignore list excludes attention, the shared experts, the dense MLP, the output head, the vision tower and the multimodal projector.[^2] Four bits for the routed experts, higher precision for everything else. Given that the routed experts are 97.9 percent of the parameters, quantising only them gets you essentially all of the compression. That is what puts a 2.8 trillion parameter checkpoint inside 1,453 GiB.

### The world it was trained in

Reinforcement learning is only as good as the environment it runs against, and about a sixth of the report is spent on environments rather than on the model. Five of them are worth describing, because together they explain where K3's agentic scores come from.

**The RL environment is a white-box agent harness.** This is the one that changed my reading most. The obvious criticism of agentic benchmarks is that they are harness-dependent, and that a vendor training against its own harness will report numbers that do not transfer to yours. Moonshot's environment is an agent harness decomposed into composable modules, tool interfaces, system prompts, context management strategies, skills, memories and subagents, which can be reassembled by configuration into Kimi Code, Claude Code, Codex, OpenClaw, Hermes or harnesses that do not exist yet. The configuration is varied across task groups specifically so the model cannot overfit to any one of them. Harness diversity is a training objective. That does not make cross-harness benchmark comparisons clean, but it does mean the obvious criticism was anticipated and engineered against rather than ignored.

**The kernel tasks come with an anti-cheating system.** K3's GPU kernel training suite spans CUDA, Triton, CuTe DSL, Gluon, ThunderKittens and TileLang across BF16, FP8 and FP4. Reward is graded rather than binary: a solution that exceeds a numerical error threshold scores zero, matching an expert implementation scores 0.5, and approaching the hardware roofline, the ceiling set by the chip's own memory bandwidth and arithmetic throughput, moves the score toward 1. What is interesting is the third component. Moonshot built a hacking-detection system that penalises the specific ways a model can fake a fast kernel, naming CUDA graph replay, input caching and precision reduction, and says they kept extending it as new tricks appeared during K3's development. The environment has an adversary in it, and the adversary is the model.

**The assistant tasks run in fake companies.** For long horizon personal-assistant work, Moonshot built mock implementations of Gmail, Notion, Slack and Canvas that preserve the real semantics without the rate limits, then wrote tasks modelled on human resources, legal and finance workflows. The agent lives in a persistent environment across multiple simulated days and meets dozens of interdependent events spread over those applications. A single rollout can run to thousands of tool calls and millions of context tokens. The starting workspaces were themselves built by agents that searched the web for reference material and assembled it into something coherent.

**The hardest environments hide the grader.** Autonomous Execution Tasks give the agent an objective, a set of constraints, a budget and a verification interface, and nothing else: no reference trajectory, no procedure. Reward comes from an independent verifier's reading of the final state of the world, not from the agent's own claim to have finished. Reward hacking is designed against directly. The agent is isolated from the verifier, a public verifier that gives diagnostic feedback is paired with a hidden one that evaluates held-out scenarios, and submissions are capped and penalised, so guessing repeatedly is expensive. The web development tasks apply the same suspicion in a simpler form: the reward is zeroed outright when a project fails to build, runs with errors, or fakes an artifact rather than implementing it.

**The vision tasks are agentic too, and that is what the tool-augmented scores are measuring.** Visual reasoning trajectories are generated inside a sandbox holding a Python interpreter. The model writes and runs code to crop, zoom or transform the image, to do exact arithmetic, or to check an intermediate result, and the outputs of that code, including images it generates, come back as new observations. It is a loop rather than a single look. This is the mechanism behind the widest gaps on the scoreboard: with Python available, ZeroBench-main goes from 23.0 to 41.0, CharXiv from 84.8 to 91.3, Math-Vision from 94.3 to 97.8. Moonshot reports that as the model learned to perform more image operations and gather more observations, its results on hard visual reasoning kept climbing. Those rows are not a model being handed a calculator. They are a model exercising a skill it was trained to have, which is why the tool-augmented column is the more honest measure of what it will do for you and the bare column is the more honest measure of what it sees.

**And the tasks build themselves.** Task synthesis runs off a self-evolving knowledge graph. Agents expand coarse seed concepts into finer ones recursively, check the existing graph for an equivalent node before adding a new one, and stop a branch once the concept is atomic enough to be a task. Tasks are then sampled from the graph, which is what lets the environment keep growing without a human writing each item.

## The shape of a conversation

K3's chat template is documented in the report's appendix, and it explains one of the model's stranger operational requirements.

The format is **XTML**, an XML-like markup with three reserved tokens for open, separator and close, plus an end-of-message token. Messages carry channels: `think`, `response`, `tools`. Which mode the model is in is determined purely by the generation prefix, not by a flag.

The consequential design decision is the ordering. Global options such as tool declarations and reasoning effort go **before** the input messages. One-shot options such as tool choice and response format go **after** them. That is not aesthetics: it means changing a per-request option does not invalidate the prefix cache for the conversation history, and it means tools can be declared mid-conversation to support dynamic tool loading. The template is laid out to protect the cache, which is to say the template is laid out to protect the price.

And **K3 only supports preserved thinking**. The `think` channel is always retained in history, even when empty. This is why the API requires you to pass the complete assistant message back, `reasoning_content` and `tool_calls` included, rather than just the content.[^7] The model was trained on histories that contain their own reasoning. Strip the reasoning and you are handing it a distribution it never saw.

The appendix also settles a documentation disagreement that existed at launch. The schema reserves four effort levels, low, medium, high and max, of which the shipped model supports a subset. Three are live today: low, high and max, defaulting to max.

## The machine underneath

Roughly a fifth of the report is infrastructure, and it is not padding. Read it and a pattern appears: nearly every architectural choice in the first half of the report bought a problem in the second half, and the fix is usually the reason the choice was affordable at all. What follows is that ledger.

**KDA needed three different kernels, not one.** A recurrent state is serial by construction, and a GPU wants wide uniform parallelism, so the mismatch shows up differently depending on what you are doing. For training and prefill, K3 uses FlashKDA, a CUTLASS kernel that overlaps the parallel work inside a chunk with the serial propagation of state between chunks, so the streaming multiprocessors are not left idle waiting for the recurrence to catch up. For prefilling a very long sequence on one device, a planner splits the sequence across the chip's own multiprocessors, evaluates each segment's transition independently, and merges them afterwards, with no cross-device traffic at all. Decoding gets a third kernel again, for reasons that turn out to be about speculation. All of it is open source, dispatched as a backend of the flash-linear-attention library.

**The prefix cache had to be rebuilt.** This is the deepest of them. Prefix caching is easy for attention, where the cache is a stack of per-token entries and any prefix of it is a valid cache. It is awkward for a recurrent state, a single blob representing everything so far and impossible to truncate. K3's solution packs KDA states into the same paged block pool as the MLA KV cache at equal page size, then decouples hash granularity from physical block size: fine 512 token hash blocks living inside much larger physical pages of 1024 to 6144 tokens, with KDA state checkpoints saved only at a sparse subset of hash boundaries, typically conversation turn boundaries. A cache hit is the longest boundary that satisfies both.

Three concurrency rules then keep it correct, and each exists because of a specific way that sharing a partly filled block can corrupt a request: pin every hit block across all cache groups before allocating anything, exclude blocks touched during the current scheduling step until their copies land, and invalidate sibling groups atomically when one group's checkpoint is evicted, so a checkpoint is hittable everywhere or nowhere. The payoff is that a hybrid model gets the same prefix caching generality a plain attention model gets, which is the reason a cached input token costs a tenth of an uncached one.

**Expert parallelism became provably balanced.** Split 896 experts across a set of machines and the router will not send them equal traffic, so some machines idle while others queue. The usual fix is to replicate the popular experts, which raises the question of how many copies are enough. MoonEP answers it: for E experts across R ranks, a perfectly balanced placement always exists with at most E/R redundant experts per rank, and the report proves both that bound and that it is essentially tight. The proof is pleasingly concrete. Repeatedly fill one underloaded rank from one overloaded rank until everything is level; each fill finishes a rank permanently, so it terminates in at most R-1 steps, and each rank ends up drawing its remote tokens from exactly one other rank. Reserve E/R slots and a feasible plan is guaranteed to exist, so training never stalls for want of one. Prior schemes cap redundancy by hand and have to stop when the cap does not fit.

Perfect balance then pays for itself twice more. Because every rank receives exactly the same number of tokens, the shapes of every computation are known ahead of time, so the per-layer synchronisation between host and device that normally stalls MoE execution can be deleted outright. And it shrinks the communication buffer from tokens times experts times ranks down to tokens times experts, since the worst case no longer has to be provisioned for. This is what the Quantile Balancing section was pointing at: a router balanced by construction is not a modelling nicety, it is what makes the fleet plannable.

**Context parallelism needed a new derivation.** The naive way to split a sequence across devices fails for KDA, because the delta rule applies a token-dependent transition to whatever state arrives, so a rank cannot compute its contribution without already knowing the state coming in, and the whole point of splitting was to avoid waiting. KDA Context Parallelism decomposes each rank's effect into two pieces that *are* locally computable: a cumulative transition that will act on whatever arrives, and a state generated from zero by the local tokens. Those pieces compose associatively, so the true incoming states are recovered with a prefix scan and a single fixed-size all-gather. The size is the good part. Splitting full attention across devices means shipping key-value blocks that grow with sequence length; splitting KDA means shipping a fixed-size state no matter how long the sequence is.

**Speculative decoding had the same problem and a different fix.** The multi-token prediction layer is fine-tuned into a draft head in the style of EAGLE-3, a standard design in which the small draft model is fed the large model's own internal features rather than only its output tokens. K3's version fuses the outputs of the first, fourth and final AttnRes blocks, low, middle and high level views of the same token. Its fusion matrix is initialised so that at step zero it reproduces exactly the high-level feature the layer was pretrained on, and learns to mix in the other two from there. It is trained against the negative log of the acceptance rate itself rather than the usual KL divergence, on the argument that for a draft model this small, matching the teacher's distribution and maximising the fraction of tokens that survive verification are not the same objective.

The KDA problem is that rejecting a speculated token means rolling back the recurrent state, and a state updated in place has nothing to roll back to. Snapshotting the state at every draft position would work, and would multiply memory traffic at the batch sizes production actually runs at. The fix turns on an observation: the state after any accepted prefix is fully determined by the drafted tokens' projected inputs, which are far smaller than the state itself. So K3 caches only those and replays the accepted tokens on chip inside a single fused kernel.

**Memory was solved by admitting the GPU is not the only memory.** Activations are quantised to FP8 and offloaded, and when a pipeline stage runs short it offloads to the memory of *other* pipeline stages over Moonshot's transfer engine rather than to the host. Gradients are sharded across data-parallel ranks and parked in CPU RAM. Between RL iterations, model weights and optimizer state are pushed out to NVMe entirely, specifically to free host DRAM for an external key-value cache pool that keeps long prefixes alive between iterations, because at a million tokens a cache miss at the start of an iteration is catastrophic. The vision encoder's wildly variable cost is hidden by scheduling most of it into pipeline bubbles.

**And the fleet has its own scheduler.** Two policies, both about predictability rather than throughput. Requests are routed to the cluster that already holds their prefix cache, since moving a 400,000 token cache between clusters is slower than recomputing it; consistent hashing assigns each session a primary and a pre-assigned secondary cluster, so a cluster failure spreads its re-prefill work across the fleet instead of dumping it on one neighbour. And because production traffic mixes 2,000 token requests with 1,000,000 token ones, a span of three orders of magnitude that makes every capacity model useless, each class of request gets its own resource budget. A burst of long-context traffic can exhaust its own share and no one else's.

**And the sandboxes.** Agentic RL needs real environments, and K3's ran on Firecracker microVMs with a checkpoint time of 133 milliseconds and a resume time of 49 milliseconds, achieved by saving only the memory pages dirtied since the last checkpoint. Because a sandbox spends as much as 98 percent of its life waiting on inference, pausing it frees all of its memory, and 6.5x memory overcommit follows. Sandboxes can be forked, so a reward judge can poke at an identical copy without disturbing the original. Moonshot notes the reason for microVMs rather than containers plainly: in early experiments, agents exploring aggressively caused kernel panics and deadlocks in container runtimes. The total across K3's training and evaluation: **51,219,741 sandboxes across 1,505,678 images.** Fifty one million disposable Linux machines is a number that reframes what training an agentic model costs, and it is the one figure of that kind the report does give.

## Does it work

> **Interactive figure:** The full K3 benchmark table from the technical report, filterable by category.

The arithmetic over the whole table: Claude Fable 5 takes 25 rows outright, K3 takes 14, GPT-5.6 Sol takes 9, one row is a tie, and Claude Opus 4.8, GPT-5.5 and GLM-5.2 win nothing at all. Head to head, K3 is ahead of Sol on 31 of the 48 rows they both ran and ahead of Fable 5 on 16 of 45. Moonshot published a table in which its own model comes second, which remains the most interesting thing about it.

The report's table is larger than the launch post's and, in places, different. Numbers moved between the two: GDPval-AA v2 from 1668 to 1686, JobBench from 52.9 to 54.3, Toolathlon from 73.2 to 76.5, Terminal-Bench 2.1's Fable 5 column from 84.6 to 88.0. And the in-house coding row split by harness, scoring 73.7 under Claude Code against 72.9 under Kimi Code, which is a useful thing to publish and an unusual thing to publish, because it puts a number on how much of an agentic score is the scaffolding rather than the model.

The other in-house result worth pulling out is blind expert judging on web development, where K3 was preferred to Claude Opus 4.8 on 58.6 percent of prompts against 27.6 percent, a 31 point margin overall that widens to 59 points on 3D and shader work.

**The internal table sorts K3 more sharply than the public one does. Presumably that is why it exists.** Sixteen in-house benchmarks, refreshed continuously to track the model's current failure modes, each model run under its own native harness except where one is forced on everybody. K3 leads Swarm Bench at 76.3 and Deep Research Bench at 90.0 by clear margins, and takes the top score on Coding Experience, a benchmark about what the model is like to work with rather than what it can solve. It trails on Agent Behavior Bench, MIRA Bench, 24/7 ClawBench, Agentic Vision Bench and KWV Bench. The shape that falls out is consistent: K3 is strongest at orchestration and research, work that decomposes into parallel pieces and is graded on a finished deliverable, and weakest at process quality and at noticing things in pictures.

One detail in that table's footnotes deserves better than a footnote. Moonshot records how often each competitor simply failed to answer. Claude Fable 5's winning 76.9 on Kimi Code Bench came with 13 fallbacks and 1 refusal across 80 tasks; it refused 14 of the Online Experience tasks and 2 on ClawBench. GPT-5.6 Sol refused 10 of 80 under Codex. K3 carries no such footnote anywhere on the table. That is obviously self-serving, and it is also a measurement almost nobody publishes, and the second thing stays true despite the first.

**Back on the public table, the evaluation configuration is where the fine print lives, and it repays reading.** Every model runs at maximum reasoning effort except GPT-5.5, which uses "xhigh". Claude Fable 5's results include its fallback behaviour and GPT-5.6 Sol's include its cyberguards, so two of the columns are measuring a shipped product rather than a bare model. Coding scores are taken under whichever of three harnesses did best for each model. Defensible, and also a reason adjacent cells in one row were not produced the same way. K3 scores 67.5 on DeepSWE here and 67.3 on the public leaderboard under a different harness. And two coding rows ran on H20 accelerators rather than the H100s those benchmarks specify. BrowseComp, K3's strongest agentic result, was run with context compaction triggered at 300K tokens rather than across the whole window. The report publishes the uncompacted million-token number as well, 90.4 against 91.2, and that small gap carries a large implication: on the model built around a million-token context, the best way to run a long task was still to throw part of it away.

That last detail matters for K3's most eye-catching coding result. The report says K3 scores 42.0 on SWE-Marathon, "7 points ahead of Claude Fable 5". That is true. It is also 2 points ahead of Claude Opus 4.8, which sits at 40.0 in the same row, and the sentence reaches past the nearest competitor to find the wider gap. Meanwhile Fable 5, the model in the seven point comparison, hit fallbacks on 35 percent of those tasks, which the configuration note discloses and the headline does not. Nothing here is fabricated and all of it is arranged, which is the standing condition of vendor benchmark tables and worth naming precisely because this vendor has otherwise been so forthcoming.

**Third party evaluation, as of 23 July 2026, lands in roughly the same place with one exception.** Artificial Analysis puts K3 at 57.1 on the v4.1 Intelligence Index, fourth of 580 models. Vals puts it second of 39. The LMArena text board puts it eighth of 200, and the agent board fourth of 37.

The exception is **WebDev Arena, where K3 is first of 99 at 1678 Elo**, which makes it the first open weights model to top that board outright. A first place in a human preference arena is a different kind of claim from a first place on a benchmark, and it is the strongest single result in the release.

One footnote on the Intelligence Index placement, because it is a good example of how a true number can still be arranged. Fourth of 580 is exactly right. Artificial Analysis' own page for K3 puts Claude Opus 5 at 60.69, Claude Fable 5 at 59.86, GPT-5.6 Sol at 58.89 and Kimi K3 at 57.11, in that order.[^9] Moonshot's table shows Fable 5 and Sol above K3, and five models below it. The model actually in first place is not on the table at all. The rank is honest and the chart around it does not contain the leader, which are both worth knowing at the same time.

**Cyber security is a new evaluation section, and it is the most consequential one.** In vulnerability discovery, roughly 70 percent of human-reviewed findings were genuine, and the model found **16 previously unknown vulnerabilities across six projects**, including a remotely triggerable heap out-of-bounds write in the Linux kernel and a Dirty-COW class privilege escalation primitive in an RDMA path. On tier-2 exploit development it solved 14 of 36 tasks against GLM-5.2's 8, on a suite the report estimates at 540 expert-hours to build and verify, roughly 15 hours per task. Moonshot then breaks its own result down in a way that undercuts it: 10 of those 14 wins are in the user-space track, and on the kernel track neither model solves three quarters of the tasks.

Set against that, an independent joint assessment by the UK AI Security Institute and NIST's CAISI reaches the same conclusion from outside. K3 beats GLM-5.2 on exploit development, 32 percent against 24 percent on ExploitBench, and completes 17 of 32 steps on a simulated enterprise network that takes a human expert about 20 hours, against GLM-5.2's 11. Against frontier cyber-capable models it trails on finishing the job, reaching **zero of 41 tasks that require arbitrary code execution**. Both things are true: it finds real bugs in real kernels, and it does not yet close the loop end to end. Publishing the external assessment alongside the internal one, including the parts that are less flattering than the internal numbers, is the correct behaviour and remains rare.

## What it costs

> **Interactive figure:** Cost per task, at list price and at measured verbosity.

The list price is thirty cents per million cached input tokens, three dollars uncached, fifteen dollars output, flat across the full million token context.[^6] The ten to one spread between a cache hit and a miss is the number that actually determines your bill, which is why the prefix caching machinery above and the option ordering in the chat template are both, in the end, pricing decisions.

The report adds a dimension the price list does not: cost per task at a fixed score. On BrowseComp, K3's best score of 91.2 percent came in at $2.03 per task, about half of GPT-5.6 Sol and an order of magnitude below Claude at maximum effort. On the coding benchmark it trails Fable 5 by four points at 38 percent of the cost, and at high effort it matches Claude Opus 4.8's maximum-effort score at roughly a third of the price.

The counterweight is verbosity. Artificial Analysis measured 130 million tokens to run their index against a 63 million median, and output tokens are the expensive half. A model that is cheap per token and generous with tokens is not automatically cheap, and reasoning effort is the lever that decides which one you get.

## The chip

The launch included a piece of evidence that is easy to file as a stunt and worth taking more seriously than that.

> **Interactive figure:** Synthesis results for nano-kpu, the accelerator K3 designed unattended.

K3 was given 48 hours with the Kimi Code harness and no human intervention, and produced **nano-kpu**: a synthesisable INT4 accelerator with fused dequantization, built against the open-source Nangate45 standard cell library, serving a scaled-down model with KDA, Block AttnRes at a block size of two, and a single shared expert under group-wise INT4 weights. Inside a 4 square millimetre area budget it closes timing at 100 MHz and simulates decoding at over 8,700 tokens per second, out of 1.46 million standard cells and 0.277 mebibytes of SRAM. The Verilog is public under Apache-2.0.

The caveats are Moonshot's own and they are real: the area and timing figures are synthesis-stage estimates that have not been through place and route, the step where designs get worse. The nano model is a smaller relative of K3, not K3.

But the shape of the thing is what matters, and it starts one stage earlier than the chip. In the kernel optimization case study K3 was set on four kernels from its own architecture, AttnRes, DeepSeek Sparse Attention, KDA, and MLA at head dimension 512, and took the AttnRes kernel from 283.6 milliseconds to 114.4, cut DSA by 55.1 percent and KDA by 73.6 percent, and got MLA past half of the chip's peak throughput. It did the work on an Nvidia Hopper part and on a second, non-Nvidia GPGPU. Across the four, it matched Claude Fable 5 running with its fallbacks available and clearly beat Claude Opus 4.8 and both GPT models. That is the capability the kernel RL environment described earlier was built to train, with its graded rewards and its anti-cheating detector.

There is a second artifact that got less attention than the chip and probably deserves more. K3 also wrote **MiniTriton**, a compact Triton-like GPU compiler: a tile-level Python frontend with its own layout system, a warp-level MLIR annotation and optimisation pass, and a PTX code generation backend, wrapped in a tensor library with reverse-mode autograd, neural network modules and NCCL distributed primitives. On an Nvidia L20 it beats both PyTorch eager and `torch.compile` in geometric mean across its own benchmark suite, and its from-scratch tensor core matmul path reaches roughly 90 percent of the measured machine roof. That is also public under Apache-2.0. A model that writes a working GPU compiler is a stranger result than a model that writes a chip, because the compiler is the thing everything else is built with.

And then there is one clause, in the same section, that is easy to read straight past. An early K3 checkpoint, Moonshot says, was already handling most of the company's kernel optimization work during K3's own late-stage development. So the model was not trained to optimise kernels and then tested on them afterwards. It was doing the job while it was being built. A model trained to optimise kernels, optimising the kernels of its own architecture, helping to build itself, and then taping out an accelerator for a miniature of itself, is not a benchmark result. It is a closed loop with one turn completed.

The other case studies extend the pattern in a different direction. A reproduction of an astrophysics result in about two hours against an estimated one to two weeks, along the way catching inconsistencies in the published formulas and writing over 3,000 lines of Python. A market analysis spanning 42 years of an industry, 120 refinement rounds, 87 quarterly reports and 99 PDFs totalling over 11,000 pages, assembled through 2,800 web searches and 1,100 terminal commands. An analysis of 391 gravitational wave events run across more than 20 concurrent subagents. And, with a directness that is either charming or ominous depending on your temperament, a 3Blue1Brown-style motion graphics explainer of its own architecture, and a launch teaser it edited itself from 56 clips.

## Where it breaks

None of that is the whole picture, and Moonshot does not pretend it is. The company published a limitations section at launch, and the technical report neither walks it back nor quietly drops it. Three admitted failures, and the report explains the mechanism behind each.

**It is sensitive to its own thinking history.** Because K3 was trained exclusively on conversations that preserve the reasoning channel, that channel is part of the input distribution rather than a byproduct of it. Drop the previous turn's reasoning to save tokens, as most chat clients do by habit, and you hand the model a shape of conversation it has never seen. An ordinary model degrades gracefully there. This one does not, and the chat template section explains why.

**It is verbose.** Measured externally, acknowledged internally, and only partly fixed. The reward model's length budget was built specifically for this and the model still generated roughly twice the median model's tokens on Artificial Analysis' index. Reasoning effort is the lever you actually have.

**Long horizon agentic runs still drift.** This is listed first by Moonshot and it is the failure the entire post-training apparatus was aimed at: the nine RL experts, the fifty one million sandboxes, the verify-in-the-loop environments, the hidden graders. All of that, and an agent running for hundreds of steps still loses the thread. It is worth sitting with the fact that this is the state of the art after that much effort, not before it.

To which the external cyber security assessment adds a fourth, from outside the company. K3 is a genuinely capable vulnerability researcher and not yet a capable exploit developer. It finds real bugs in real kernels, and yet on Moonshot's own exploit suite ten of its fourteen wins are in user space, on the kernel track it fails three quarters of the tasks, and on the external assessment it completes zero of 41 tasks that require turning a bug into arbitrary code execution. Those are different lines to be standing on, and the distance between them is the distance between a defensive research tool and an offensive one.

## What is still not published

> **Interactive figure:** Thirty-one things a reader might want to know about K3, split into what the release answered and what it did not.

The gaps that remain are consistent, and they are all the same gap wearing different clothes. **How many tokens it was trained on. On what hardware. For how long. At what cost.** Everything about how the model is shaped is public. Almost nothing about what it took to shape it is. The two accidental disclosures noted earlier, the few hundred GPUs and the H20s, are as close as the document comes, and neither appears in a section that intends to tell you anything.

There is one absence that is easy to miss because it looks like a presence. The contributions appendix lists roughly four hundred names, alphabetically by surname, with no roles, no teams and no ordering by seniority. As a refusal of the authorship politics that dominate this field it is admirable. As data it is also the only real measure the report gives of what building one of these takes, and it is a headcount rather than a budget.

One structural omission is worth knowing if you plan to serve this yourself. The report describes an EAGLE-3 style draft head in detail, including its fusion matrix, its initialisation and its loss function, and the report's own comparison table lists K3 as carrying one multi-token prediction layer, the same as K2. The shipped `config.json` sets that count to zero.[^2] Speculative decoding is documented, tabulated, and not in the box.

## The license

K3 is open weights, and it is not the license K2 had.

K2 shipped under a Modified MIT license whose only real teeth were an attribution clause. The **Kimi K3 License** keeps that clause, requiring prominent display of "Kimi K3" for deployments above 100 million monthly active users or $20 million in monthly revenue, and adds a new one.[^8]

The new clause is about Model-as-a-Service. If you operate a MaaS business with more than $20 million in aggregate revenue over any consecutive 12 month period, you must enter a separate agreement with Moonshot before any commercial use of K3. There are carve-outs for internal use and for access through Moonshot's own products and certified partners.

The target is unambiguous. Individuals, researchers and companies serving K3 for their own purposes are unaffected. Large inference providers reselling it are now negotiating. This is the first open frontier weights release I am aware of to draw that line explicitly, and it is probably not the last, because it is the line that decides whether releasing weights is a marketing expense or a revenue channel.

## The read

The architecture is three changes, each of which sounds small.

A scalar gate became a vector, so a fixed size memory can forget selectively. A sum over layers became a softmax, so depth became something the model chooses rather than something it endures. A feed-forward layer moved into a compressed latent space, so the expert count could nearly triple. Sequence, depth, width.

What makes K3 worth reading closely is not the ideas, which were published months earlier in models nobody deployed. It is that they were carried all the way through. The decay floor was chosen so a Tensor Core instruction would not overflow. The activation has a ceiling of exactly 100 because two shape parameters multiply to it, and the ceiling is smooth so gradients survive touching it. The chat template's option ordering was chosen so the prefix cache would survive. The quantization scheme covers the 98 percent of parameters where it pays and nothing else, and was applied during training rather than after it. The expert placement is provably balanced so a synchronisation point could be deleted. Every one of those is an architecture decision and a deployment decision at the same time, and the report is unusual in refusing to pretend they are separable.

Against that, the thing still missing from the record is money. We know what K3 is down to the config file and how it works down to the kernel, and we do not know what it took to build. That is the shape of the last remaining secret in open weights releases, and it is telling that it is the one nobody gives up.

**Glossary**

- **AttnRes**: Attention Residuals. Instead of every layer's output entering the residual stream with weight one, each layer takes a learned softmax over the outputs beneath it and reads a weighted mixture. The weights sum to one, so depth stops inflating the stream. K3 uses the block variant, which attends over groups of twelve layers rather than individual ones.
- **harness**: The scaffolding around a model during an agentic benchmark: how tools are exposed, how errors are retried, how context is compacted, how many turns are allowed. It is a large fraction of measured agent ability, which is why comparing scores across different harnesses compares two things at once.
- **KDA**: Kimi Delta Attention, K3's linear attention layer. It keeps a fixed size state matrix, erases along the incoming key before writing, and decays what it holds at a separate learned rate for every feature channel, so it can forget some things faster than others. 69 of K3's 93 layers are KDA.
- **KV cache**: The stored keys and values for every token seen so far, kept so that generating the next token does not require recomputing the whole sequence. It grows linearly with context, and at a million tokens it dominates the memory bill. 69 of K3's 93 layers do not keep one.
- **linear attention**: An attention layer that keeps a fixed size state matrix instead of a growing cache of past keys and values. Cost per token stays constant no matter how long the sequence gets, which is the point, and the state can only hold so much, which is the price.
- **MLA**: Multi-head Latent Attention. Full attention that compresses keys and values through a low rank bottleneck before caching them, so the cache shrinks without changing what the layer computes. K3 adds a gate to it and removes positional encoding from it.
- **MoE**: Mixture of Experts. The feed-forward layer is split into many separate experts and a router picks a few per token, so total parameters and per-token compute are decoupled. K3 takes 16 of 896, and those experts are 98 percent of the model.
- **Muon**: A matrix aware optimizer that orthogonalizes gradient updates rather than treating weights as a flat vector of scalars. Moonshot has used it since K2 and extends it in K3 to orthogonalize each attention head's block independently.
- **MXFP4**: A four bit floating point format in which every group of 32 values shares one exponent scale. K3 stores its routed experts this way and was trained that way from supervised fine-tuning onward, rather than being quantized after the fact.
- **NoPE**: No Position Encoding. A layer given no explicit signal about where in the sequence each token sits. K3 applies it to every full attention layer, which leaves KDA's decay as the only thing in the model that knows about order.
- **on-policy distillation**: Training a student on its own generations, scored by a teacher, rather than on transcripts the teacher wrote. The student stays inside the distribution it will actually occupy at inference, which is the standing failure mode of the offline kind.
- **prefix cache**: Reusing the computed state of a shared prompt prefix across requests, so the same system prompt is not paid for twice. Straightforward for standard attention, where the cache is a stack of per-token entries. Awkward for a recurrent state, which is why KDA needed a purpose-built implementation.
- **QAT**: Quantization-aware training. Running the low precision format during training rather than compressing a finished model afterwards, so the weights are learned in the format they will be served in and there is no accuracy cliff at the end.
- **speculative decoding**: Generating several tokens at once with a small cheap draft model and then checking them all in a single pass of the large one. Accepted tokens are close to free. Rejected ones cost a rollback, which is the part a recurrent state makes hard.

[^1]: Kimi Team, "Kimi K3 Technical Report", [`MoonshotAI/Kimi-K3`](https://github.com/MoonshotAI/Kimi-K3), 27 July 2026. 47 pages, released with the weights. Source for the three-axis framing, the KDA decay bound and its Tensor Core motivation, Block AttnRes, Stable Latent MoE, SiTU-GLU, Quantile Balancing, MoonViT-V2's from-scratch pretraining, Per-Head Muon, the pretraining corpus and its programmatic multimodal component, the scaling law and schedule comparison, the full post-training pipeline, all infrastructure sections, the public and in-house benchmark tables with their configuration notes, the cost-efficiency tables, the cyber security evaluation, the case studies, and the chat template appendix. It is PDF-only and is not on arXiv.

[^2]: [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) on Hugging Face, weights and `config.json` published 27 July 2026. Source for every architectural number quoted here: 93 layers, hidden dimension 7168, 96 attention heads, the 69 KDA and 24 Gated MLA split with the extra final full-attention layer, 896 routed experts, 16 active, 2 shared, latent MoE dimension 3584, per-expert hidden dimension 3072, `attn_res_block_size` 12, NoPE on MLA, the MXFP4 quantization group and its ignore list, and `num_nextn_predict_layers` set to zero. The checkpoint is 1,453 GiB across 96 safetensors shards.

[^3]: Moonshot AI, ["Kimi K3"](https://www.kimi.com/blog/kimi-k3), 16 July 2026. The launch post. Its benchmark table is superseded by the report's, and several figures were revised between the two. Its Kernel Arena widget quotes 96 layers and model dimension 8192, which the config shows is not K3's shape, so that widget describes a benchmark configuration rather than the production model.

[^4]: Kimi Team, ["Kimi Linear: An Expressive, Efficient Attention Architecture"](https://arxiv.org/abs/2510.26692), arXiv:2510.26692, 30 October 2025. Source for the four-way comparison of update rules, the FLOP crossover, the 3:1 ratio ablation and the NoPE argument. Its flagship model is 48B-A3B. K3's KDA differs from it in the decay parameterization and the rank of the output gate, both documented in the K3 report.

[^5]: Kimi Team, ["Attention Residuals"](https://arxiv.org/abs/2603.15031), arXiv:2603.15031, 16 March 2026. Source for the AttnRes equations, the depth-mixing analysis and the output-magnitude result. Its model uses 6 layers per block against K3's 12, and its architecture section describes KDA interleaved with full attention at three to one, which is what makes it the first model to run both of K3's headline mechanisms together.

[^6]: Moonshot AI platform pricing, `platform.kimi.ai`. K3 at $0.30 cache-hit input, $3.00 cache-miss input and $15.00 output per million tokens, with a 1,048,576 token context and no tiering by context length. The parameter breakdown in "What it actually is" is my own arithmetic over `config.json`, and reproduces the report's totals to 2.776T against a stated 2.78T and 100.5B against a stated 104.2B, the remainder being components the config does not fully specify.

[^7]: Kimi API reference and the report's chat template appendix. `reasoning_effort` accepts low, high and max and defaults to max; thinking cannot be disabled. Preserved thinking requires the complete assistant message, including `reasoning_content` and `tool_calls`, to be passed back unmodified. Evaluation used top-p 0.95 for single-step reasoning and vision and top-p 1.0 for agentic tasks.

[^8]: Kimi K3 License, shipped in the Hugging Face repository. Not the Modified MIT license K2 used. The Model-as-a-Service clause applies above $20 million in aggregate revenue over any consecutive 12 month period; the attribution clause applies above 100 million monthly active users or $20 million in monthly revenue.

[^9]: Artificial Analysis, `artificialanalysis.ai/models/kimi-k3`. The comparison series embedded in that page reads, in order: Claude Opus 5 (max) 60.6919, Claude Fable 5 (with fallback) 59.8606, GPT-5.6 Sol (max) 58.8898, Kimi K3 57.1123, Grok 4.5 (high) 53.8266, GLM-5.2 (max) 51.0858. Moonshot's Table 5 quotes 57.1, 59.9 and 58.9 for the three it lists, which match to the decimal.