Agent Skills
› rmyndharis/antigravity-skills
› ml-pipeline-workflow
ml-pipeline-workflow
GitHub提供端到端MLOps流水线构建指南，涵盖数据准备、模型训练、验证及生产部署。支持DAG编排、自动化工作流设计、版本控制与监控，适用于从零搭建ML系统或优化现有MLOps实践。
Trigger Scenarios
构建新的机器学习流水线
设计ML系统的工作流编排
实现数据到模型部署的自动化
设置可重复的训练工作流
创建基于DAG的ML编排
Install
npx skills add rmyndharis/antigravity-skills --skill ml-pipeline-workflow -g -y
SKILL.md
Frontmatter
{
"name": "ml-pipeline-workflow",
"description": "Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows."
}
ML Pipeline Workflow
Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.
Do not use this skill when
- The task is unrelated to ml pipeline workflow
- You need a different domain or tool outside this scope
Instructions
- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
Overview
This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.
Use this skill when
- Building new ML pipelines from scratch
- Designing workflow orchestration for ML systems
- Implementing data → model → deployment automation
- Setting up reproducible training workflows
- Creating DAG-based ML orchestration
- Integrating ML components into production systems
What This Skill Provides
Core Capabilities
-
Pipeline Architecture
- End-to-end workflow design
- DAG orchestration patterns (Airflow, Dagster, Kubeflow)
- Component dependencies and data flow
- Error handling and retry strategies
-
Data Preparation
- Data validation and quality checks
- Feature engineering pipelines
- Data versioning and lineage
- Train/validation/test splitting strategies
-
Model Training
- Training job orchestration
- Hyperparameter management
- Experiment tracking integration
- Distributed training patterns
-
Model Validation
- Validation frameworks and metrics
- A/B testing infrastructure
- Performance regression detection
- Model comparison workflows
-
Deployment Automation
- Model serving patterns
- Canary deployments
- Blue-green deployment strategies
- Rollback mechanisms
Reference Documentation
See the references/ directory for detailed guides:
- data-preparation.md - Data cleaning, validation, and feature engineering
- model-training.md - Training workflows and best practices
- model-validation.md - Validation strategies and metrics
- model-deployment.md - Deployment patterns and serving architectures
Assets and Templates
The assets/ directory contains:
- pipeline-dag.yaml.template - DAG template for workflow orchestration
- training-config.yaml - Training configuration template
- validation-checklist.md - Pre-deployment validation checklist
Usage Patterns
Basic Pipeline Setup
# 1. Define pipeline stages
stages = [
"data_ingestion",
"data_validation",
"feature_engineering",
"model_training",
"model_validation",
"model_deployment"
]
# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example
Production Workflow
-
Data Preparation Phase
- Ingest raw data from sources
- Run data quality checks
- Apply feature transformations
- Version processed datasets
-
Training Phase
- Load versioned training data
- Execute training jobs
- Track experiments and metrics
- Save trained models
-
Validation Phase
- Run validation test suite
- Compare against baseline
- Generate performance reports
- Approve for deployment
-
Deployment Phase
- Package model artifacts
- Deploy to serving infrastructure
- Configure monitoring
- Validate production traffic
Best Practices
Pipeline Design
- Modularity: Each stage should be independently testable
- Idempotency: Re-running stages should be safe
- Observability: Log metrics at every stage
- Versioning: Track data, code, and model versions
- Failure Handling: Implement retry logic and alerting
Data Management
- Use data validation libraries (Great Expectations, TFX)
- Version datasets with DVC or similar tools
- Document feature engineering transformations
- Maintain data lineage tracking
Model Operations
- Separate training and serving infrastructure
- Use model registries (MLflow, Weights & Biases)
- Implement gradual rollouts for new models
- Monitor model performance drift
- Maintain rollback capabilities
Deployment Strategies
- Start with shadow deployments
- Use canary releases for validation
- Implement A/B testing infrastructure
- Set up automated rollback triggers
- Monitor latency and throughput
Integration Points
Orchestration Tools
- Apache Airflow: DAG-based workflow orchestration
- Dagster: Asset-based pipeline orchestration
- Kubeflow Pipelines: Kubernetes-native ML workflows
- Prefect: Modern dataflow automation
Experiment Tracking
- MLflow for experiment tracking and model registry
- Weights & Biases for visualization and collaboration
- TensorBoard for training metrics
Deployment Platforms
- AWS SageMaker for managed ML infrastructure
- Google Vertex AI for GCP deployments
- Azure ML for Azure cloud
- Kubernetes + KServe for cloud-agnostic serving
Progressive Disclosure
Start with the basics and gradually add complexity:
- Level 1: Simple linear pipeline (data → train → deploy)
- Level 2: Add validation and monitoring stages
- Level 3: Implement hyperparameter tuning
- Level 4: Add A/B testing and gradual rollouts
- Level 5: Multi-model pipelines with ensemble strategies
Common Patterns
Batch Training Pipeline
# See assets/pipeline-dag.yaml.template
stages:
- name: data_preparation
dependencies: []
- name: model_training
dependencies: [data_preparation]
- name: model_evaluation
dependencies: [model_training]
- name: model_deployment
dependencies: [model_evaluation]
Real-time Feature Pipeline
# Stream processing for real-time features
# Combined with batch training
# See references/data-preparation.md
Continuous Training
# Automated retraining on schedule
# Triggered by data drift detection
# See references/model-training.md
Troubleshooting
Common Issues
- Pipeline failures: Check dependencies and data availability
- Training instability: Review hyperparameters and data quality
- Deployment issues: Validate model artifacts and serving config
- Performance degradation: Monitor data drift and model metrics
Debugging Steps
- Check pipeline logs for each stage
- Validate input/output data at boundaries
- Test components in isolation
- Review experiment tracking metrics
- Inspect model artifacts and metadata
Next Steps
After setting up your pipeline:
- Explore hyperparameter-tuning skill for optimization
- Learn experiment-tracking-setup for MLflow/W&B
- Review model-deployment-patterns for serving strategies
- Implement monitoring with observability tools
Related Skills
- experiment-tracking-setup: MLflow and Weights & Biases integration
- hyperparameter-tuning: Automated hyperparameter optimization
- model-deployment-patterns: Advanced deployment strategies
Version History
- e63f7dd Current 2026-07-05 09:35
