用相对游标分页

When requesting multiple pages of records from a server the simple implementation is to have an incremental page number in the URL. Starting at page one, each subsequent request that’s sent has a page number that’s one greater than the previous. The problem is that incremental page numbers scale poorly—the bigger the page number, the slower the query. The simple solution is relative cursor pagination because it remembers where you were and continues from that point onwards instead.

当从服务器上请求多页记录时,简单的实现是在URL中设置一个递增的页码。从第一页开始,每个后续发送的请求都有一个比前一个大的页码。问题是,增量页码的扩展性很差--页码越大,查询速度越慢。简单的解决方案是相对光标分页,因为它能记住你的位置,并从该点开始继续。

The Problem

问题所在

A common activity for third-party applications on Shopify to do is to sync the full catalogue of products. Some shops have more than 100,000 products and these can’t all be loaded in a single request as it would time out. Instead, the application would make multiple requests to Shopify for successive pages of products which look like this:

Shopify上的第三方应用程序的一个常见活动是同步全部产品目录。有些商店有超过100,000种产品,这些产品不能在一次请求中全部加载,因为会超时。相反,应用程序会向Shopify发出多个请求,以获取连续的产品页面,看起来像这样。

https:.myshopify.com/admin/products.json?page=25&limit=100

https:.myshopify.com/admin/products.json?page=25&limit=100

This would generate a SQL query like this:

这将产生一个这样的SQL查询。

This query scales poorly because the bigger the offset, the slower the query. In the above example, the query needs to go through 2500 records and then discard the first 2400. Using a test shop with 14 million products, we ran some experiments loading pages of products at various offsets. Taking the average time over five runs at each offset, here are the results:

这种查询的扩展性很差,因为偏移量越大,查询的速度就越慢。在上面的例子中,该查询需要浏览2500条记录,然后丢弃前2400条。我们使用一个有1400万个产品的测试商店,在不同的偏移量下进行了一些加载产品页面的实验。以每个偏移量的五次运行的平均时间为例,以下是结果。

Offset

偏移

Time (ms)

时间(毫秒

10

10

6.54

6.54

100

100

7.72

7.72

1,000

1,000

8.46

8.46

10,000

10,000

79.82

79.82

100,000

100,000

2,221.60

2,221.60

Omitted from the table are tests with the 1,000,000th offset and above since they consistently timed out.

表中省略了第1,000,000次偏移量及以上的测试,因为它们总是超时。

Not only do queries take ...

开通本站会员,查看完整译文。

首页 - Wiki
Copyright © 2011-2026 iteam. Current version is 2.148.4. UTC+08:00, 2026-01-25 15:32
浙ICP备14020137号-1 $访客地图$