用Ratchet处理ETL任务
At Grab, the Lending team is focused towards building products that help finance various segments of users, such as Passengers, Drivers, or Merchants, based on their needs. The team builds products that enable users to avail funds in a seamless and hassle-free way. In order to achieve this, multiple lending microservices continuously interact with each other. Each microservice handles different responsibilities, such as providing offers, storing user information, disbursing availed amounts to a user’s account, and many more.
在 Grab，借贷团队专注于打造产品，帮助不同阶层的用户，如乘客、司机或商人，根据他们的需求进行融资。该团队打造的产品能够让用户以无缝和无障碍的方式利用资金。为了实现这一目标，多个借贷微服务不断地相互作用。每个微服务处理不同的职责，如提供优惠、存储用户信息、向用户的账户支付利用的金额，等等。
In this tech blog, we will discuss what Data and Extract, Transform and Load (ETL) pipelines are and how they are used for processing multiple tasks in the Lending Team at Grab. We will also discuss Ratchet, which is a Go library, that helps us in building data pipelines and handling ETL tasks. Let’s start by covering the basis of Data and ETL pipelines.
在这个技术博客中，我们将讨论什么是数据和提取、转换和加载（ETL）管道，以及它们如何用于处理Grab公司贷款团队的多项任务。我们还将讨论 棘轮，它是一个Go库，可以帮助我们建立数据管道和处理ETL任务。让我们先来了解一下数据和ETL管道的基础。
What is a Data Pipeline?
什么是数据管道？
A Data pipeline is used to describe a system or a process that moves data from one platform to another. In between platforms, data passes through multiple steps based on defined requirements, where it may be subjected to some kind of modification. All the steps in a Data pipeline are automated, and the output from one step acts as an input for the next step.
数据管道是用来描述一个系统或一个将数据从一个平台转移到另一个平台的过程。在各平台之间，数据根据定义的要求通过多个步骤，在这些步骤中，数据可能会被进行某种修改。数据管道中的所有步骤都是自动化的，一个步骤的输出可以作为下一个步骤的输入。
Data Pipeline (Source: Hazelcast)
数据管道（来源：Hazelcast）。
An ETL pipeline is a type of Data pipeline that consists of 3 major steps, namely extraction of data from a source, transformation of that data into the desired format, and finally loading the transformed data to the destination. The destination is also known as the sink.
ETL管道是一种数据管道，由3个主要步骤组成，即从源头提取数据，将数据转换为所需格式，最后将转换后的数据加...