我们如何将CI YAML文件从1800行减少到50行

This article illustrates how the Cauldron Machine Learning (ML) Platform team uses GitLab parent-child pipelines to dynamically generate GitLab CI files to solve several limitations of GitLab for large repositories, namely:

本文说明了Cauldron机器学习（ML）平台团队如何使用GitLab父子管道来动态生成GitLab CI文件，以解决GitLab对大型存储库的几个限制，即。

• Limitations to the number of includes (100 by default).
• 对包括的数量的限制（默认为100）。
• Simplifying the GitLab CI file from 1800 lines to 50 lines.
• 将GitLab的CI文件从1800行简化到50行。
• Reducing the need for nested gitlab-ci yml files.
• 减少了对嵌套的gitlab-ci yml文件的需求。

Introduction

简介

Cauldron is the Machine Learning (ML) Platform team at Grab. The Cauldron team provides tools for ML practitioners to manage the end to end lifecycle of ML models, from training to deployment. GitLab and its tooling are an integral part of our stack, for continuous delivery of machine learning.

Cauldron 是 Grab 的机器学习（ML）平台团队。Cauldron团队为ML从业者提供工具，以管理ML模型的端到端生命周期，从训练到部署。GitLab和它的工具是我们堆栈的一个组成部分，用于持续交付机器学习。

One of our core products is MerLin Pipelines. Each team has a dedicated repo to maintain the code for their ML pipelines. Each pipeline has its own subfolder. We rely heavily on GitLab rules to detect specific changes to trigger deployments for the different stages of different pipelines (for example, model serving with Catwalk, and so on).

我们的核心产品之一是MerLin管道。每个团队都有一个专门的 repo 来维护他们的 ML 管道的代码。每个管道都有自己的子文件夹。我们在很大程度上依靠GitLab规则来检测特定的变化，以触发不同管道的不同阶段的部署（例如，用Catwalk的模型服务，等等）。

Background

背景介绍

Approach 1: Nested child files

方法1：嵌套的子文件

Our initial approach was to rely heavily on static code generation to generate the child gitlab-ci.yml files in individual stages. See Figure 1 for an example directory structure. These nested yml files are pre-generated by our cli and committed to the repository.

我们最初的方法是严重依赖静态代码生成，在各个阶段生成子gitlab-ci.yml 文件。参见图1的目录结构示例。这些嵌套的yml文件是由我们的clipper预先生成的，并提交到存储库中。

Figure 1: Example directory structure with nested gitlab-ci.yml files.

Child gitlab-ci.yml files are added by using the include keyword.

通过使用include关键字添加子gitlab-ci.yml 文件。

Figure 2: Example root .g...