홈 항목 library logo 글 코드베이스 상점
자세히
类库 › kimi-k3-in-c
FareedKhan-dev

FareedKhan-dev/kimi-k3-in-c

这是一个在纯C99环境下运行的Kimi K3大模型推理引擎。无需GPU、BLAS或外部框架，仅用单CPU和8GB内存即可运行2.78万亿参数模型。代码轻量便携，旨在展示极致优化能力，主要面向Linux x86-64平台。

6,581 1,072 6,581 5
在 GitHub 上查看
FareedKhan-dev/kimi-k3-in-c

标签

avx2 c99 cpu-inference deep-learning from-scratch inference-engine kimi-k3 linear-attention llm llm-inference machine-learning memory-efficient mixture-of-experts moe mxfp4 quantization simd systems-programming transformer zero-dependencies

截图

A small resident working set on top, the model itself on NVMe underneath, and a few labelled pipes between them
Four steps from a server cluster down to an ordinary laptop, with the same output at both ends
What each preset actually costs in memory
The naive requirement: every parameter resident at bf16
One token wakes 16 experts and leaves 880 asleep
Only 16 of 896 experts fire per layer, so most of the model sleeps
Where the 1.56 TB lives: 93% of it is experts that never load
The always-active set: 113.49 GB at bf16, everything else is streamable
Four reductions, and the output is identical at both ends
One binary, four kinds of machine, and one identical answer
A 176 KB binary that runs a 1.56 TB model
safetensors: one length, one header, then raw bytes at known offsets
Index the shard, read the exact bytes on demand, then drop the pages
96 shards, 1,560,936,091,448 bytes, verified one file at a time
One-based layer indices, and 92 and 93 are both MLA by design
Refuse rather than guess, because a guessed field gives you a different model
Every case goes through a file, never through argv
Four files in, byte-identical files back out
MXFP4: a 4-bit nibble scaled by one 8-bit exponent per 32 weights
One byte carries two weights, and the low nibble is the even one
Half a byte per weight plus the shared scale gives one expert exactly
The low nibble is the EVEN element, and reversing it is silently wrong
What dequantizing would cost, which is why we multiply from the nibbles
Half a byte per weight saves 4 TB, and never dequantizing saves 194 GB a token
RMSNorm with epsilon inside the square root, accumulated in double
A fixed reduction order, so scalar and AVX2 agree bit for bit
Same weights, three code paths, one hash
Where one token goes on the floor configuration: 80% of it is waiting on disk
The recurrent state is the same size at 10 tokens and at 100,000
Every token folds into the same fixed-size state
Why 69 layers are KDA: its state does not grow with context
Decay the state, read from it, write the delta, then read the updated state
A depthwise causal convolution of width 4 with the activation fused in
A sum of squares, not a mean, and applied to q and k only
The decay gate, with A indexed per head and not per channel
The delta rule: decay, read, write the difference, then read again
Nine ordered steps, and the numbering is not decoration
Norm first, then gate, then project, and that order is not interchangeable
One latent per position is cached, and k and v are rebuilt on use
The query and the token both pass through a small shared latent
Twenty-four MLA layers, expanded k and v in fp32, per position
What MLA caches per position, per layer
The released code caches expanded heads, the latent is 53x smaller
Each layer attends over the outputs of every preceding block
Blocks of twelve, so the residual stack never exceeds nine sources
Every twelve layers the running prefix is snapshotted and cleared
One layer: aggregate, attend, aggregate again, then route
Attention residuals drawn by the data: activations climb, then collapse every 12 layers
The bias steers selection only, the weights come from unbiased scores
Experts run in a narrow latent, and the norm is on the aggregate
Route, run 16 experts in a 3584-wide latent, then project back up
SiTU-GLU with beta1 = 4 and beta2 = 25, so the product is bounded
The hottest experts, out of 10,010 distinct ones touched
Refuse if the bytes are not contiguous, because a gap means copying experts
O_DIRECT needs both ends aligned, which costs 4 KB per layer
Packing 93 contiguous layer runs into one 108.81 GB file
The path one token takes, from cold NVMe to the next word
Pinned layers cost nothing to read, everything else comes through one ring slot
Everything is summed before anything is allocated, then compared to free RAM
One token: embed, walk 93 layers, aggregate, project to the vocabulary
A fixed walk order means the next read can start before this layer finishes
Streaming turns a 315 GB floor into an 11 GB dial
A slot is empty, reserved but not yet readable, or holding an expert
The expert cache holds whole experts, so the budget divides exactly
Reserve serially, read in parallel, then publish only what arrived
Batched preads keep the device busy, serial gets leave it idle
An eightfold spread in storage speed, and the engine is I/O bound
One run, one trace, then replay it at every capacity
Every distinct expert must be read once, so no policy can beat this
The experts this trace touched at all, against a 1.45 TB pool
The lever is the policy, not the size: LRU is flat where Belady climbs
Four levels of proof, and only the last two touch the released checkpoint
The rounding budget for 93 layers at hidden size 7168
Op fixtures, a toy oracle, then the released checkpoint
All 93 layers checked against torch: 93 passed, 0 failed, 5639 seconds
Compare all 163,840 logits, not just the winner
Prompt in, 93 layers, one argmax, one word out
Seven memory ceilings, seven different speeds, one identical answer
One prefill wall, then a steady shelf, on every prompt
Prefill scales with the prompt, and nothing after it does
One prefill wall, then a steady shelf, on all four prompts
Prefill scales with the prompt, and nothing after it does
Cap the memory with a cgroup, then check the ids are identical
28x the memory buys 1.70x the speed, and most steps are inside the noise
The memory plan is not lying: every rung lands on its own budget
Token zero pays the pinning cost, so short runs understate the hit rate
The expert cache does nothing at all until about 36 GB of arena
The simulation was wrong in both directions
One counter, three answers, and only one of them means avoided I/O
Three definitions of one number, and only the third agrees with the bytes
The only expert-cache metric that agrees with the bytes actually read
What a working expert cache actually looks like, step by step
The trunk is re-read in full every token, the experts are only sampled
What one pinned layer removes from the guaranteed traffic
Give the trunk everything until it is pinned, then feed the cache
At a fixed budget, giving the trunk more is 1.69x faster
The winner reads 79% more expert bytes and still wins
Same binary, same prompt, same flags, same machine, same minute
Three identical runs, so a smaller difference is not an effect
Only two timing effects clear the band
The same configuration, measured twice, doing identical work
This is an I/O problem at every budget, from 8 GB to 224 GB
Both terms are whole-run totals, and this is an I/O problem at every budget
The same code on two devices, and the gap is the device
Symmetric per-row quantization, the method behind the int8 and int4 study
No layer type tolerates 4 bits any better than the others
int8 costs about 1%, int4 costs about 17%, and its worst rows cost 65%
Seconds are bought back with RAM, and rounding error is not, at any budget
The advertised million-token context is a memory fact, not an engine limit
What this engine does not implement, drawn to scale
From a 1.56 TB checkpoint to the right answer, on one machine
docs/images/patrick_pray.png

相关项目

sammycage
sammycage/plutofilter
PlutoFilter是一个单头文件、零内存分配的图像滤镜库，快速应用各种图像效果，兼容SVG和CSS语义。
194 C
tectonic-typesetting
tectonic-typesetting/tectonic
Tectonic是一个现代化、完整的自包含TeX/LaTeX引擎，基于XeTeX和TeXLive，旨在简化文档编译过程。
4.3k C
pgvector
pgvector/pgvector
pgvector是一个用于Postgres的开源向量相似性搜索工具，可实现精确和近似最近邻搜索。支持多种向量格式和距离计算，兼具Postgres的ACID特性。
17.4k C
FelixKratz
FelixKratz/SketchyBar
SketchyBar是一个高度可定制的macOS状态栏替代品，旨在提供快速、灵活和强大的状态栏体验，支持脚本和动态配置。
9.9k C
TEN-framework
TEN-framework/ten-vad
TEN VAD是一个低延迟、高性能且轻量级的语音活动检测器，适用于各种语音处理应用。
1.4k C
tree-sitter
tree-sitter/tree-sitter-html
tree-sitter-html是用于Tree-sitter的HTML语法解析库，主要帮助开发者解析和分析HTML语言。
179 C

评论

홈 - 위키
Copyright © 2011-2026 iteam. Current version is 2.155.2. UTC+08:00, 2026-08-31 10:37
浙ICP备14020137号-1 $방문자$