FareedKhan-dev/kimi-k3-in-c - herramientas en línea

herramientas en línea

herramientas en línea

反馈

herramientas en línea

biblioteca biblioteca de código tienda

más

类库 › kimi-k3-in-c

FareedKhan-dev/kimi-k3-in-c

这是一个在纯C99环境下运行的Kimi K3大模型推理引擎。无需GPU、BLAS或外部框架，仅用单CPU和8GB内存即可运行2.78万亿参数模型。代码轻量便携，旨在展示极致优化能力，主要面向Linux x86-64平台。

5,779 938 5,779 18

在 GitHub 上查看

标签

avx2 c99 cpu-inference deep-learning from-scratch inference-engine kimi-k3 linear-attention llm llm-inference machine-learning memory-efficient mixture-of-experts moe mxfp4 quantization simd systems-programming transformer zero-dependencies

截图

Half a byte per weight plus the shared scale gives one expert exactly

Half a byte per weight saves 4 TB, and never dequantizing saves 194 GB a token

Where one token goes on the floor configuration: 80% of it is waiting on disk

Decay the state, read from it, write the delta, then read the updated state

A depthwise causal convolution of width 4 with the activation fused in

A sum of squares, not a mean, and applied to q and k only

Nine ordered steps, and the numbering is not decoration

Norm first, then gate, then project, and that order is not interchangeable

One latent per position is cached, and k and v are rebuilt on use

Twenty-four MLA layers, expanded k and v in fp32, per position

The released code caches expanded heads, the latent is 53x smaller

Each layer attends over the outputs of every preceding block

Blocks of twelve, so the residual stack never exceeds nine sources

Every twelve layers the running prefix is snapshotted and cleared

Attention residuals drawn by the data: activations climb, then collapse every 12 layers

The bias steers selection only, the weights come from unbiased scores

Route, run 16 experts in a 3584-wide latent, then project back up

Refuse if the bytes are not contiguous, because a gap means copying experts

O_DIRECT needs both ends aligned, which costs 4 KB per layer

Packing 93 contiguous layer runs into one 108.81 GB file

The path one token takes, from cold NVMe to the next word

Pinned layers cost nothing to read, everything else comes through one ring slot

Everything is summed before anything is allocated, then compared to free RAM

One token: embed, walk 93 layers, aggregate, project to the vocabulary

A fixed walk order means the next read can start before this layer finishes

Streaming turns a 315 GB floor into an 11 GB dial

A slot is empty, reserved but not yet readable, or holding an expert

Reserve serially, read in parallel, then publish only what arrived

Batched preads keep the device busy, serial gets leave it idle

The experts this trace touched at all, against a 1.45 TB pool

The rounding budget for 93 layers at hidden size 7168

Op fixtures, a toy oracle, then the released checkpoint

All 93 layers checked against torch: 93 passed, 0 failed, 5639 seconds

Prompt in, 93 layers, one argmax, one word out

Seven memory ceilings, seven different speeds, one identical answer

One prefill wall, then a steady shelf, on every prompt

Prefill scales with the prompt, and nothing after it does

One prefill wall, then a steady shelf, on all four prompts

Prefill scales with the prompt, and nothing after it does

Cap the memory with a cgroup, then check the ids are identical

28x the memory buys 1.70x the speed, and most steps are inside the noise

The memory plan is not lying: every rung lands on its own budget

Token zero pays the pinning cost, so short runs understate the hit rate

The expert cache does nothing at all until about 36 GB of arena

The simulation was wrong in both directions

One counter, three answers, and only one of them means avoided I/O

The only expert-cache metric that agrees with the bytes actually read

What a working expert cache actually looks like, step by step

What one pinned layer removes from the guaranteed traffic

Give the trunk everything until it is pinned, then feed the cache

At a fixed budget, giving the trunk more is 1.69x faster

The winner reads 79% more expert bytes and still wins

Same binary, same prompt, same flags, same machine, same minute

Three identical runs, so a smaller difference is not an effect

Only two timing effects clear the band

The same configuration, measured twice, doing identical work

The same code on two devices, and the gap is the device

Symmetric per-row quantization, the method behind the int8 and int4 study

No layer type tolerates 4 bits any better than the others

int8 costs about 1%, int4 costs about 17%, and its worst rows cost 65%

Seconds are bought back with RAM, and rounding error is not, at any budget

The advertised million-token context is a memory fact, not an engine limit

What this engine does not implement, drawn to scale

From a 1.56 TB checkpoint to the right answer, on one machine

docs/images/patrick_pray.png

相关项目

评论

项目信息

语言: C
许可证: Apache License 2.0
默认分支: main
所有者: FareedKhan-dev User
创建时间: 2026-08-01
最近提交: 1周前
最近更新: 22小时前
收录时间: 2026-08-09

支持平台

Linux

inicio - Wiki
Copyright © 2011-2026 iteam. Current version is 2.155.2. UTC+08:00, 2026-08-17 06:19
浙ICP备14020137号-1 $mapa de visitantes$