trang chủ chủ đề library logo thư viện mã nguồn cửa hàng
thêm
类库 › kimi-k3-in-c
FareedKhan-dev

FareedKhan-dev/kimi-k3-in-c

这是一个在纯C99环境下运行的Kimi K3大模型推理引擎。无需GPU、BLAS或外部框架，仅用单CPU和8GB内存即可运行2.78万亿参数模型。代码轻量便携，旨在展示极致优化能力，主要面向Linux x86-64平台。

5,061 810 5,061 16
在 GitHub 上查看
FareedKhan-dev/kimi-k3-in-c

标签

avx2 c99 cpu-inference deep-learning from-scratch inference-engine kimi-k3 linear-attention llm llm-inference machine-learning memory-efficient mixture-of-experts moe mxfp4 quantization simd systems-programming transformer zero-dependencies

截图

The query and the token both pass through a small shared latent
Twenty-four MLA layers, expanded k and v in fp32, per position
What MLA caches per position, per layer
The released code caches expanded heads, the latent is 53x smaller
Each layer attends over the outputs of every preceding block
Blocks of twelve, so the residual stack never exceeds nine sources
Every twelve layers the running prefix is snapshotted and cleared
One layer: aggregate, attend, aggregate again, then route
Attention residuals drawn by the data: activations climb, then collapse every 12 layers
The bias steers selection only, the weights come from unbiased scores
Experts run in a narrow latent, and the norm is on the aggregate
Route, run 16 experts in a 3584-wide latent, then project back up
SiTU-GLU with beta1 = 4 and beta2 = 25, so the product is bounded
The hottest experts, out of 10,010 distinct ones touched
Refuse if the bytes are not contiguous, because a gap means copying experts
O_DIRECT needs both ends aligned, which costs 4 KB per layer
Packing 93 contiguous layer runs into one 108.81 GB file
The path one token takes, from cold NVMe to the next word
Everything is summed before anything is allocated, then compared to free RAM
One token: embed, walk 93 layers, aggregate, project to the vocabulary
A fixed walk order means the next read can start before this layer finishes
Streaming turns a 315 GB floor into an 11 GB dial
A slot is empty, reserved but not yet readable, or holding an expert
The expert cache holds whole experts, so the budget divides exactly
Reserve serially, read in parallel, then publish only what arrived
Batched preads keep the device busy, serial gets leave it idle
One run, one trace, then replay it at every capacity
Every distinct expert must be read once, so no policy can beat this
The experts this trace touched at all, against a 1.45 TB pool
The lever is the policy, not the size: LRU is flat where Belady climbs
Four levels of proof, and only the last two touch the released checkpoint
The rounding budget for 93 layers at hidden size 7168
Op fixtures, a toy oracle, then the released checkpoint
All 93 layers checked against torch: 93 passed, 0 failed, 5639 seconds
Compare all 163,840 logits, not just the winner
Prompt in, 93 layers, one argmax, one word out
Seven memory ceilings, seven different speeds, one identical answer
One prefill wall, then a steady shelf, on every prompt
Prefill scales with the prompt, and nothing after it does
One prefill wall, then a steady shelf, on all four prompts
Prefill scales with the prompt, and nothing after it does
Cap the memory with a cgroup, then check the ids are identical
28x the memory buys 1.70x the speed, and most steps are inside the noise
The memory plan is not lying: every rung lands on its own budget
Token zero pays the pinning cost, so short runs understate the hit rate
The expert cache does nothing at all until about 36 GB of arena
The simulation was wrong in both directions
One counter, three answers, and only one of them means avoided I/O
Three definitions of one number, and only the third agrees with the bytes
What a working expert cache actually looks like, step by step
The trunk is re-read in full every token, the experts are only sampled
What one pinned layer removes from the guaranteed traffic
Give the trunk everything until it is pinned, then feed the cache
At a fixed budget, giving the trunk more is 1.69x faster
The winner reads 79% more expert bytes and still wins
Same binary, same prompt, same flags, same machine, same minute
Three identical runs, so a smaller difference is not an effect
Only two timing effects clear the band
The same configuration, measured twice, doing identical work
This is an I/O problem at every budget, from 8 GB to 224 GB
Both terms are whole-run totals, and this is an I/O problem at every budget
The same code on two devices, and the gap is the device
Symmetric per-row quantization, the method behind the int8 and int4 study
Seconds are bought back with RAM, and rounding error is not, at any budget
The advertised million-token context is a memory fact, not an engine limit
From a 1.56 TB checkpoint to the right answer, on one machine
docs/images/patrick_pray.png

相关项目

sammycage
sammycage/plutofilter
PlutoFilter是一个单头文件、零内存分配的图像滤镜库，快速应用各种图像效果，兼容SVG和CSS语义。
194 C
tectonic-typesetting
tectonic-typesetting/tectonic
Tectonic是一个现代化、完整的自包含TeX/LaTeX引擎，基于XeTeX和TeXLive，旨在简化文档编译过程。
4.3k C
pgvector
pgvector/pgvector
pgvector是一个用于Postgres的开源向量相似性搜索工具，可实现精确和近似最近邻搜索。支持多种向量格式和距离计算，兼具Postgres的ACID特性。
17.4k C
FelixKratz
FelixKratz/SketchyBar
SketchyBar是一个高度可定制的macOS状态栏替代品，旨在提供快速、灵活和强大的状态栏体验，支持脚本和动态配置。
9.9k C
TEN-framework
TEN-framework/ten-vad
TEN VAD是一个低延迟、高性能且轻量级的语音活动检测器，适用于各种语音处理应用。
1.4k C
tree-sitter
tree-sitter/tree-sitter-html
tree-sitter-html是用于Tree-sitter的HTML语法解析库，主要帮助开发者解析和分析HTML语言。
179 C

评论

trang chủ - Wiki
Copyright © 2011-2026 iteam. Current version is 2.155.2. UTC+08:00, 2026-08-16 16:05
浙ICP备14020137号-1 $bản đồ khách truy cập$