利用 SLM 结合边缘设备构建 AIoT Agent

1. 利用 SLM 结合边缘设备构建 AIoT Agent 卢建晖微软高级云技术布道师

2.

3. 模型发展

4. 模型继续发展文本向量语音视频图片闭源模型 – 云端开源模型 – 本地

5. 企业最终选择 RAG 行业数据微调混合模型 Local SLMs Cloud LLM 云 + 本地的算力架构 GPU NPU CPU

6. SLM

7. 什么是 SLM (Small Language Model) LLM SLM

8. 厂商的 SLM 角力

9. 模型选择 – LLM vs SLM 性能 1. 更少的算力需求 2. 部署在更小的设备甚至边缘计算场景上无障碍 1. 更多开发者和组织可以使用 2. 具备一定的业务能力，便于企业开发人员使用定制化 1. 针对特定领域和任务进行微调 2. 所有权

10. 不同的开源小模型应用场景图片，代码 LLaVA AudioCraft NanoVLM 向量模型 NanoDB jina-embeddings-v2-base-cn 知识/文本/聊天 Qianwen-chat OpenELM Gemma-2b Phi3

11. 影响构建行业模型的四大要素数据部署评估算法 • 如何组织数据 • Qlora vs Lora • 模型性能 • 边缘设备的部署 • 企业内部多个数据源的整理 • 参数调整 • 提示工程有效性 • 模型压缩 • 计算 • 效能 • 回应 • 本地或云端 • 存储 • 迭代 • • 如何有效组织数据数据安全

12. Azure AI Azure AI Studio One place for building and deploying AI solutions Model Benchmarking Azure AI Infrastructur e State-of-the-art silicon and systems for AI workloads Continuous Monitoring Code-First Experience Azure AI Services Pre-trained, turnkey solutions for intelligent applications Azure OpenAI Service Azure AI Search Azure AI Speech Azure AI Vision Azure AI Content Safety Azure AI Document Intelligence Azure AI Language Azure AI Translator High-Bandwidth Networking Azure Machine Learning Full-lifecycle tools for designing and managing responsible AI models Microfluidic Cooling Azure Maia Silicon Solution Evaluation Responsible AI Model Catalog Prompt Flow Orchestration Cutting-Edge Models Model Fine Tuning Access to the latest foundation and open-source models Model Training

13. SLM + Azure ML Prompt flow Microsoft Fabric ONNX Runtime • Unified data platform • Result Evaluation • Cross-platform • Lake House can build data based on business • Compare models • Response • business flow integration • Int4, float32, float16 multi- format compatible • Cloud + local data integration Datastore Prompt Flow Orchestration Deployment Azure Machine Learning Full-lifecycle tools for designing and managing responsible AI models Compute Nvidia Model Catalog

14. SLM Ops SLM SLM Fine-tuning Model Evaluation dataset Azure Machine Learning / Azure AI Studio Full-lifecycle tools for designing and managing responsible AI models Deployment

15. Microsoft Olive

16. Microsoft Olive Microsoft Olive是一款非常易用的开源模型优化工具，既可以涵盖Generative AI领域的微调，也可以引用。只需简单的配置，结合使用开源SLM和相关运行时环境（AzureML/本地GPU、CPU、DirectML），通过自动优化即可完成模型的微调或参考，让您找到最佳模型部署到云或边缘设备。 Windows AI Studio 上引入了 Microsoft Olive microsoft/Olive: Olive is an easy-to-use hardware-aware model optimization tool that composes industry-leading techniques across model compression, optimization, and compilation. (github.com) Microsoft Olive makes SLMOps engineering easier for development teams

17. Microsoft Olive - Making the SLMOps process easy

18. 混合解决方案 SLM SLM Fine-tuning Model Evaluation dataset Compute SLMOps Endpoint Deployment

19. Olive.config as Service

20. Olive.config – Model Catalog

21. Olive.config - Compute Settings

22. Olive.config – Qlora & Lora

23. Olive.config – Convert ONNX Model Convert （Float16，Float32，Int4）

24. Provides a base for the Copilot Stack

25. Demo

26.

27.

28. Running Phi3 in Jetson + +

29. Running Phi3 in AIPC

30. Running Phi3 in Mobile

31.

32.