Scaling Nextdoor’s Datastores: Part 2

In the second installment of Nextdoor’s “Scaling Nextdoor’s Datastores” blog series, the Core-Services team discusses challenges faced after implementing database read replicas.

Adding read replicas to an existing database is a very common pattern as applications or products evolve to handle increased demand. Typically, the implementation details are hand waved and it’s assumed that this strategy will work. However, that is rarely the case, and we’ll dive into some more of the intricacies around the implementation.

Initial Attempt

When replicas were first introduced in the Nextdoor stack, we gave the product engineers latitude to choose when they wanted to have their query routed to a read replica or to the primary. This was done by leveraging the existing routing mechanism in our ORM, Django.

This seemed like the right idea at the time because the product engineers had the most context around consistency requirements within their changes and load characteristics of their product feature. Therefore, they would have the best ability to judge which node to send their query to. However, as our business logic evolved and became more feature-rich, product engineers began to add abstraction layers to help abstract complex operations away from business logic.

In this design evolution there is a high frequency read, followed by a low frequency conditional write, followed by a read. The read performed after the write should be routed to the primary, but that may get buried in abstractions and this requirement regressed.

The explicit routing decisions engineers made became buried and subsequently created a serious problem for users of these abstractions. If one abstraction method was performing a write and another a read, they could not safely be used together due to read-after-write consistency issues. Due to replication lag between the primary and replica databases, a race condition arises when the application attempts to read data from a replica after performing a write.

We had created a system where engineers had to be aware of the entire call stack and be able to determine if this situation would apply to them or not. The easiest way to handle this situation? Always use the primary…

The Band-aid

A common solution engineers employed was to wrap higher-order business logic in database transactions because within the context of a transaction, all queries are routed to the primary.

Shows that the choice to route to primary is made in the abstraction layer so functions A, B, and C all use the primary whether they need to or not.

Author’s Note: Our default isolation level for transactions, repeatable read, gave engineers a false sense of security, as it only guarded against replication lag and not racing writes with concurrent transactions. We have since improved this to ensure read-your-own-write semantics.

This strategy had a negative effect on database load because it indiscriminately caused all queries intended to be sent to replica databases to be sent to the primary database. The impact of this problem increased as:

1. more business logic leveraged the database

2. business logic increased its use of existing abstractions

3. query performance decreased as more data was added

What transpired was a years-long erosion of the capacity benefits the additional read replicas initially provided. As a result, the load on the primary database node became one of the most pressing issues with Nextdoor’s database stacks.

It was clear that the initial approach of exposing routing choices to engineers was no longer tenable and the team embarked on a way of making the replica vs. primary decision for the product engineers.

Reimagining

Using ORMs (Object Relation Mappings) is controversial. There are many pros and cons and we won’t debate all of them here. However, one of the advantages of an ORM is that there is a consistent layer of abstraction between the database and the application. This allowed us to inject a simple piece of custom logic to keep track of which tables have been written to while processing a web request. Why is it helpful to keep track of modified tables? By doing this we could automatically make informed decisions of where to route subsequent read queries, regardless of where they were buried in the business logic stack.

This simple strategy, coupled with our already high read-to-write ratio, allowed us to shift much of the read traffic to the replica databases and substantially reduce our reliance on the primary database.

Author’s Note: While the strategy was simple, we did have to cleanup up all of the manual routing decisions along with inappropriate usages of transactions.

While this naive approach was rather effective, we realized that this strategy was actually too conservative. We noticed that our average database replication lag was around 20ms while our web requests lasted an order of magnitude longer. That means that even after the update had been replicated to the primary, we were still disallowing queries for that table to the read replica. This provided an opportunity to use a timing based system that marked tables as re-eligible after p99.9 replication lag had elapsed. With this additional optimization, we were able to re-route most of the queries from our primary to the read replicas.

Takeaway

Each application will have its own eccentricities that make RDBMS scalability a challenge for platform teams but we hope this post provides a cautionary tale, as well as a potential solution for similar cases.

In the next post, Appropriately serializing data for caching, we’ll cover how we serialize database data for caching and some of the pitfalls to be aware of when introducing caching to an application.