评估从 Scylla 支持的服务中移除 Redis-cache 对性能的影响

At Grab, we operate a set of services that manage and provide counts of various items. While this may seem straightforward, the scale at which this feature operates—benefiting millions of Grab users daily—introduces complexity. This feature is divided into three microservices: one for “writing” counts, another for handling “read” requests, and a third serving as the backend for a portal used by data scientists and analysts to configure these counters.

在Grab，我们运营一组服务，管理和提供各种项目的计数。虽然这看起来很简单，但这个功能的规模——每天惠及数百万Grab用户——引入了复杂性。这个功能分为三个微服务：一个用于“写入”计数，另一个用于处理“读取”请求，第三个作为数据科学家和分析师用于配置这些计数器的门户的后端。

This article focuses on the service responsible for handling “read” requests. This service is backed by Scylla storage and a Redis cache. It also connects to a MySQL RDS to retrieve “counter configurations” that are necessary for processing incoming requests. Written in Rust, the service serves tens of thousands of queries per second (QPS) during peak times, with each request typically being a “batch request” requiring multiple lookups (~10) on Scylla.

本文重点介绍负责处理“读取”请求的服务。该服务由Scylla存储和Redis缓存支持。它还连接到MySQL RDS以检索处理传入请求所需的“计数器配置”。该服务使用Rust编写，在高峰时段每秒处理数万次查询（QPS），每个请求通常是一个“批量请求”，需要在Scylla上进行多次查找（约10次）。

Recently, the service has encountered performance challenges, causing periodic spikes in Scylla QPS. These spikes occur throughout the day but are particularly evident during peak hours. To understand this better, we’ll first walk you through how this service operates, particularly how it serves incoming requests. We will then explain our proposed solution and the outcomes of our experiment.

最近，服务遇到了性能挑战，导致Scylla QPS周期性激增。这些激增在一天中都会发生，但在高峰时段尤为明显。为了更好地理解这一点，我们将首先向您介绍该服务的运作方式，特别是它如何处理传入请求。然后我们将解释我们提出的解决方案及实验结果。

Anatomy of a request

请求的结构

Each counter configuration stored in MySQL has a template that dictates the format of incoming queries. For example, this sample counter configuration is used to count the raindrops for a specific city:

存储在MySQL中的每个计数器配置都有一个模板，规定了传入查询的格式。例如，这个示例计数器配置用于计算特定城市的雨滴数量：

{
    "id": 34,
    "name": "count_rain_dro...