22580: 从GPT2到Kimi3，详解

[

user avatar

](https://x.com/waterloo_intern)

ali

[

Baseten

](https://twitter.com/baseten)

Article cover image

22580: From GPT2 to Kimi3, Explained

22580：从 GPT2 到 Kimi3 详解

Twenty-two thousand five hundred and eighty. That’s how many GPT-2 (2019) models fit inside KimiK3 (2026). We scaled up by a factor of 22,580 in seven years. But is it just... scale?

两万两千五百八十。这就是 KimiK3 (2026) 内部能容纳的 GPT-2 (2019) 模型的数量。我们在七年内将规模扩大了 22,580 倍。但这仅仅是……规模吗？

In this worklog, I’ll walk through how we got here and how much, or how little, has actually changed since then. We’ll trace the major architectural developments leading to KimiK3.

在这篇工作日志中，我将带大家了解我们是如何走到这一步的，以及自那以后究竟发生了多少变化，或者说变化有多小。我们将追溯引领至 KimiK3 的主要架构发展脉络。

GPT-2

GPT-2 is a decoder-only architecture:

GPT-2 是一种 decoder-only architecture：

tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd)
pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd)
x = self.transformer.drop(tok_emb + pos_emb)
for block in self.transformer.h:
    x = block(x)
x = self.transformer.ln_f(x)
logits = self.lm_head(x)
return logits

The input receives token and positional embeddings:

输入接收 token 和 positional embeddings：

Each transformer block, zoomed in, looks like this:

放大后的每个 transformer block 看起来是这样的：

class Block(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.ln_1 = LayerNorm(config.n_embd, bias=config.bias)
        self.attn = CausalSelfAttention(config)
        self.ln_2 = LayerNorm(config.n_embd, bias=config.bias)
        self.mlp = MLP(config)

    def forward(self, x):
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

The attention process:

注意力过程：

B, T, C = x.size() # batch size, sequence length, embedding dimensionality (n_embd)

        # calculate query, key, values for all heads in batch and move head forward to be the batch dim
        q, k, v  = self.c_attn(x).split(self.n_embd, dim=2)
        k = k.view(B, T, self.n_head, C /...