Agent Skills
› sediman-agent/OpenSkynet
› huggingface-local-models
huggingface-local-models
GitHub用于在本地通过llama.cpp运行GGUF模型。支持搜索Hugging Face仓库、选择量化版本、启动llama-cli/server服务及OpenAI兼容接口，并处理模型转换与认证。
Trigger Scenarios
用户希望使用llama.cpp在本地运行LLM
需要查找或下载Hugging Face上的GGUF格式模型
请求配置本地推理服务器以兼容OpenAI API
询问关于GGUF量化格式的选择建议
Install
npx skills add sediman-agent/OpenSkynet --skill huggingface-local-models -g -y
SKILL.md
Frontmatter
{
"name": "huggingface-local-models",
"description": "Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving."
}
Hugging Face Local Models
Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.
Default Workflow
- Search the Hub with
apps=llama.cpp. - Open
https://huggingface.co/<repo>?local-app=llama.cpp. - Prefer the exact HF local-app snippet and quant recommendation when it is visible.
- Confirm exact
.gguffilenames withhttps://huggingface.co/api/models/<repo>/tree/main?recursive=true. - Launch with
llama-cli -hf <repo>:<QUANT>orllama-server -hf <repo>:<QUANT>. - Fall back to
--hf-repoplus--hf-filewhen the repo uses custom file naming. - Convert from Transformers weights only if the repo does not already expose GGUF files.
Quick Start
Install llama.cpp
brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make
Authenticate for gated repos
hf auth login
Search the Hub
https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
Run directly from the Hub
llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
Run an exact GGUF file
llama-server \
--hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
--hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
-c 4096
Convert only when no GGUF is available
hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
--outfile model-f16.gguf \
--outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
Smoke test a local server
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"messages": [
{"role": "user", "content": "Write a limerick about exception handling"}
]
}'
Quant Choice
- Prefer the exact quant that HF marks as compatible on the
?local-app=llama.cpppage. - Keep repo-native labels such as
UD-Q4_K_Minstead of normalizing them. - Default to
Q4_K_Munless the repo page or hardware profile suggests otherwise. - Prefer
Q5_K_MorQ6_Kfor code or technical workloads when memory allows. - Consider
Q3_K_M,Q4_K_S, or repo-specificIQ/UD-*variants for tighter RAM or VRAM budgets. - Treat
mmproj-*.gguffiles as projector weights, not the main checkpoint.
Load References
- Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.
- Read quantization.md for format tables, model scaling, quality tradeoffs, and
imatrix. - Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.
Resources
- llama.cpp:
https://github.com/ggml-org/llama.cpp - Hugging Face GGUF + llama.cpp docs:
https://huggingface.co/docs/hub/gguf-llamacpp - Hugging Face Local Apps docs:
https://huggingface.co/docs/hub/main/local-apps - Hugging Face Local Agents docs:
https://huggingface.co/docs/hub/agents-local - GGUF converter Space:
https://huggingface.co/spaces/ggml-org/gguf-my-repo
Version History
- c9d8953 Current 2026-07-05 19:55
