Data scientists at Shopify expect fast results when querying large datasets across multiple data sources. We use Trino (a distributed SQL query engine) to provide quick access to our data lake and recently, we’ve invested in speeding up our query execution time.

On top of handling over 500 Gbps of data, we strive to deliver p95 query results in five seconds or less. To achieve this, we’re constantly tuning our infrastructure. But with each change comes a risk to our system. A disruptive change could stall the work of our data scientists and interrupt our engineers on call.

That’s why Shopify’s Data Reliability team built custom verification, benchmarking, and profiling tooling for testing and analyzing Trino. Our tooling is designed to minimize the risk of various changes at scale.

Below we’ll walk you through how we developed our tooling. We’ll share simple concepts to use in your own Trino deployment or any other complex system involving frequent iterations.

An overview of how we investigated and solved the issue of some Kubernetes Pods running MySQL starting up and shutting down slower than other similar Pods with the same data set.

At Shopify, we develop a bunch of different React Native mobile apps: Shop, Inbox, Point of Sale, Shopify Mobile, and Local Delivery. These apps represent different business domains, but they often have shared pieces of functionality like login or foundational blocks they build upon. Wouldn’t it be great to leverage development speed and focus on important product features by reusing code other teams have already written? Sure, but it might be a big and time consuming effort that discourages teams. Usually, contributing to a new repo is more tedious and error prone in comparison to contributing to an existing repository. The developer needs to create a new repository, set up continuous integration (CI) and distribution pipelines, and add configs for Jest, ESint, and Babel. It might be unclear where to start and what to do.

My team, React Native Foundations, decided to invest in simplifying the process for developers at Shopify. In this post, I'll walk you through the process of extracting those shared elements, the setup we adopted, the challenges we encountered, and future lines of improvement.

Shopify’s infrastructure supports over 1.7 million merchants during their entrepreneurship journey. A key component of the current infrastructure is the underlying fleet of MySQL database shards that together persist every shops’ critical data. As traffic patterns change and new merchants onboard onto the platform, it’s possible that resource intensive shops end up living in the same shards. Certain database shards become unbalanced in their database utilization, shop traffic, and load. It’s important to ensure the shards remain well-balanced to mitigate risk of database failure, improve productivity of the wider infrastructure, and ultimately guarantee buyers can always access their favorite shops. This post explains how we’re able to balance our MySQL shards by migrating shops across shards—entirely online and with virtually zero consumer-facing downtime.

We built a custom write-through cache for the Shop app's home feed that reduced database load by 15% and overall app latency by about 20%.

We reevaluated our existing product categorization model to ensure we’re understanding what our merchants are selling, to build the best products that help power their sales.

Mental models within an industry, company, or even a person, change constantly. As methodologies mature, we see the long term effects our choices have wrought and can adjust accordingly. As a team or company grows, methodologies that worked well for five people may not work as well for 40 people. If all employees could keep an entire app in their head, we’d need fewer rules and checks and balances on our development, but that is not the case. As a result, we summarize things we notice have been implicit in our work.

Rate limiting is a system that protects the stability of APIs. GraphQL opens new possibilities for rate limiting. I’ll show you Shopify’s rate limiting system for the GraphQL Admin API and how it addresses some limitations of common methods commonly used in REST APIs. I’ll show you how we calculate query costs that adapt to the data clients need while providing a more predictable load on servers.

Using incremental page numbers for pagination scales poorly, so to solve this, Shopify uses relative cursor pagination to deliver faster queries.

In the previous post about our new storefront rendering engine, we described how we went about the rewrite process and smoothly transitioned to serve storefront requests with the new implementation. As a follow-up and based on readers’ comments and questions, this post dives deeper into the technical details of how we built the new storefront rendering engine to be faster than the previous implementation.

To set the table, let’s see how the new storefront rendering engine performs:

• It generates a response in less than ~45ms for 75% of storefront requests;
• It generates a response in less than ~230ms for 90% of storefront requests;
• It generates a response in less than ~900ms for 99% of storefront requests.

Thanks to the new storefront rendering engine, the average storefront response is nearly 5x faster than with the previous implementation. Of course, how fast the rendering engine is able to process a request and spit out a response depends on two key factors: the shop’s Liquid theme implementation, and the number of resources needed to process the request. To get a better idea of where the storefront rendering engine spends its time when processing a request, try using the Shopify Theme Inspector: this tool will help you identify potential bottlenecks so you can work on improving performance in those areas.

At Shopify, we’re keeping the flexibility of untrusted Partner code, but executing it on our own infrastructure with WebAssembly.

对于中小商家，SaaS企业会提供标准化的产品，价格较为便宜；也有针对大企业的解决方案，比如Shopify提供Shopify Plus，2000美金一个月起.....

好的 leadership 至少不应该让团队更多的感受到不确定性，尽管在外部世界，不确定性已经是一种常态。从外部到内部，leadership 应该发挥一种过滤器的作用。吸纳、处理、判断、沟通。从内部看，方向和目标都应该是明确的，即便发生变化，也是井然有序，而非突然来袭。

Shopify是全球电商SaaS龙头，为各类零售商搭建电商独立站点，管理全渠道的营销、售卖、支付、物流、仓储等一站式服务。公司由Tobias Lutke于2004年创办，总部位于加拿大首都渥太华，前身是一家二手滑板线上零售网站Snowdevil，2006 年转型为SaaS服务商，2015 年实现美股上市。目前公司是为全球规模最大的电商类SaaS 提供商，员工人数达到5000人，业务涵盖美国，英国，加拿大，澳大利亚等175个国家/地区拥有超过100万的企业客户，其中包括Allbirds、Gymshark、PepsiCo、Staples等大型品牌客户。

Shopify发展路线是从网站搭建工具到平台再到电商生态系统，是典型的企业级服务公司发展路线。Shopify基于软件工具，提供包括网站安全、页面设计、销售数据统计分析在内的全方位功能，扩展多元化业务建立属于自己的生态。目前Shopify平台与Amazon、Ebay 等电商合作，同时登陆Facebook、Instagram、Pinterest、Google+等主流社交媒体，已经建立了自己的完整电商生态系统。

最近，Bessmer Ventures Partners（BVP）公开了它们非常成功的几个投资案例的投资备忘录，其中包括 Shopify、LinkedIn、Pinterest、Twillio、Yelp 等。

而 Shopify 这个项目，是 Alex Ferrara 在 2010 年担任 Bessemer 新合伙人时做的第一个案子，当时他主导了 Bessemer 对 Shopify 的 A 轮投资。

比较有挑战的是，Shopify 的 CEO 那时还非常年轻，并且是第一次创业，所以 Alex Ferrara 说他当时非常紧张。如今 10 年过去了，Shopify 已经成为加拿大最有价值的公司。

从这些投资备忘录，我们能看到这些伟大的投资在当时的决策逻辑和背景。

Shopify的口号是“arming the rebels”，为小电商们提供网上开店的软件，对抗Amazon邪恶帝国。而事实是，小电商之间相互竞争，Shopify更像是军火商，不站边只赚钱。