In Part 4: Keeping the cache consistent, we highlighted a class of consistency issues arising from racing cache writes and introduced an approach for forward cache versioning as a mechanism to avoid inconsistencies. The cache is able to decide which write to persist and which to reject because it is aware of the version of data it currently has. However, this is only a partial solution because it assumes writers will always succeed in communicating with the cache in a timely manner, if at all.

Missed Writes

Let’s consider the scenario where Writers A and B both performed an update to the same row in the database and have not yet updated the cache. Writer A holds Version 1 and Writer B holds Version 2. What happens if Writer B with Value 2 fails to talk with the cache?

In this case the result is that the cache becomes inconsistent and we can’t rely on the writers to provide that consistency. A process must exist outside of this interaction to fix-up the cache when Version 2 is written to the database but fails to be written to the cache.

Change Data Stream

To solve this problem we tap into a common feature provided by most modern databases, a Change Data Capture (CDC) Stream. A CDC Stream is a mechanism to subscribe to row level changes in a database.

The change stream contains a row’s previous column values along with the new values. Here’s a visual example of the change stream when the last_name field gets updated in the database.

For visual clarity the changed values have been underlined in red.

Reconciler

Since the database is the source of truth and the CDC Stream emits all changes, a consumer of this stream can clean up any consistency issues in the cache. In our system we call this process reconciliation and it’s performed by the ”Reconciler.”

The reconciler is responsible for converting CDC change information into input for the del_if_version call to the cache. The arguments for the del_if_version are the version and cache key.

As discussed in part 4, del_if_version performs conditional deletion of data in the cache (as evaluated by the cache itself) if the supplied version is less than the provided version.

Author’s Note: Having both old and new column values is particularly important when your cache key is comprised of row values or if you have secondary cache keys for unique column values.

For the above example the del_if_version function call would have parameters version=2 and key=lastnames:1. The key parameter is made up of two parts, the table name (lastnames) and the primary key for the specific record (row 1).

Reconciling Missed Cache Writes

Having covered the building blocks, let’s reconsider the case of a missed write. In the below diagram we show that the Reconciler is able to delete Version 1 from the cache in the event Writer B does not succeed in updating the cache with Version 2.

Author’s Note: Astute readers will point out that we could have avoided this issue if the database change stream was used to update the cache. The caveat to this approach is that if you want to keep a look-aside cache design, you then must maintain two different methods of serialization (from ORM objects and from the raw change stream) in sync.

Missed Write During Cache Fill

Thus far we’ve only discussed writers doing database updates and then updating the cache. However, anytime a reader checks the cache and gets a miss, the reader must populate the cache after fetching from the database.

The last scenario to consider is what happens when a reader is populating the cache due to a cache miss while a writer is performing a write? Depending on the timing of the sequence of steps the cache may become inconsistent. Consider the following:

1. Due to a cache miss the Reader reads Version 1 from the database
2. The Writer reads Version 1 from the database, performs business logic, and writes Version 2 to the database
3. The Writer fails to write Version 2 to the cache
4. The Reconciler receives Version 2 and deletes Version 1 from the cache
5. After some time the Reader re-populates Version 1 in the cache

The end state is the cache has Version 1 and is now inconsistent with the database.

Reconciling Missed Write During Cache Fill

The solution to this consistency problem is conceptually simple. The Reconciler must lag behind such that Step 4 in the above diagram happens after Step 5. However, a problem arises in that until the reconciler deletes the stale version in the cache, the cache remains inconsistent.

For our desired consistency level we wanted to have the Reconciler fix-up the most commonly occurring issues as quickly as possible. In practice missing writes are extremely uncommon for us.

A simple solution to this problem is having two reconcilers. One instance is reading the change stream and applying fixes to the cache in near real time while another one is always a fixed amount of time behind performing any final mopping up. In our case this delay was slightly higher than our web request timeout.

Author’s Note: There are a handful of other scenarios where a cache inconsistency may occur, but those are fairly complex to articulate. We encourage readers to think through the cases of incomplete and out of order steps. Readers will find that the two-pass reconcilation handles those cases.

Reconciliation Pipeline Implementation

The discussion about reconciliation has, until now, been theoretical, without addressing the key implementation details and design goals. One functional requirement was to guarantee that users have a cohesive experience, such as seeing their own writes reflected after a page refresh. This meant that reconciliation needed to happen in near real time to provide a seamless experience in the event of a missed cache write.

The choice to use row versions for conditional deletion of cache keys has a subtle but important consequence: changes do not need to be processed in order. This conscious design choice meant that we could horizontally scale our processing in order to satisfy low latency reconciliation.

The Reconciler, which we have only discussed conceptually until now, is actually made up of three different pieces.

Pg-bifrost

Pg-bifrost, an open-source tool previously developed by our team, consumes the PostgreSQL WAL Replication log (CDC Stream) and republishes it for use in other applications. It was built with low latency republishing in mind.

Apache Kafka

Kafka was selected as our persistent message bus for the WAL change stream from the database due to its low latency, persistence guarantees, and well-supported consumer and producer APIs. As Nextdoor was already utilizing Kafka, it was a natural choice for our team.

Reconciler

The reconciler is simply a GoLang based Kafka consumer that reads the WAL change stream and executes Redis del_if_version calls. The two-pass reconciliation, discussed earlier, was implemented using a time wheel to maintain a fixed offset while remaining relatively straightforward.

Putting It All Together

This series began by exploring typical relational database scalability issues and the questions teams must consider when addressing them. Part 2 presented an approach to improve read replica usage with semi-intelligent routing. Part 3 discussed the importance of serializing cached data and its impact on scalability. Part 4 outlined a cache consistency strategy using row versions and conditional upserts. The final installment, Part 5, highlighted consistency issues and profiled a reconciliation system to maintain database-cache consistency.

This multifaceted strategy solved the following scalability challenges for our datstores:

• Reduced the load on our primary PostgresSQL databases
• Ensured that only forward versions of database rows were cached for a limited time
• Prevented thundering herds to the cache and databases when database schemas changed
• Made better use of read replicas, which enable effective horizontal scalability for RDBMS

Future Work

This project was a stepping stone on our database scalability journey. Keen readers likely noticed that our caching system is limited to retrieving items by their primary key or unique attributes. This is due to the fact that key-value storage in a cache is best suited for these types of lookups, and they are essentially an extension of unique database indexes.

The next step in our database scalability journey is to enable the storage of lists within the cache. This enhancement will allow us to efficiently handle common database queries, such as “Give me a list of people in a neighborhood”.

Parting Thoughts From the Authors

Slava Markeyev: Technologies like no-SQL and new-SQL are often thought of as magic pill solutions to scalability challenges. While they have many advantages over traditional RDBMS solutions, a datastore can fundamentally only be as scalable as the schema and query patterns allow it to be. Changing those two after the fact is akin to rebuilding the engine while the plane is in the air. Doing so is certainly possible — heck, we tried it — but that is a story for another day.

My parting thought for readers is SQL databases when used in conjunction with secondary indexes like Redis/Valkey for caching and ElasticSearch for search will get you pretty far if you let them.

Tushar Singla: While doing architecture interviews, the need often arises to consider how to scale the database to support 100M or even 1B+ users. A common strategy is to “scale out” or add “caching.” Typically, mentioning those concepts are enough to communicate understanding of these horizontally scaling techniques and the interview moves on. However, only after building out this system in real life at scale can one appreciate the intricacies, complexities, and myriad of edge cases that must be handled in order to serve the users appropriately.

Next time when you mention these techniques during the interview, take some time to consider what it might look like to actually finish the implementation.

Ronak Shah: Scaling a datastore presents unique challenges, and there is no one-size-fits-all solution. Finding the right approach for your system requires thoughtful discussion and a rigorous design process. Although our work was complex and challenging, it was also a truly rewarding experience to be part of.

P.S. Having robust observability tools and a solid unit testing framework can save countless engineering hours spent debugging cache consistency issues.

Scaling Nextdoor’s Datastores: Part 5 was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, we’ll focus specifically on inconsistencies caused by racing writes and our solution. We’ll discuss other causes and our full solution for consistent caching in the next installment of our blog: Part 5: A time-bounded eventually-consistent cache.

Inconsistent Cache

Caches can become inconsistent with the database for several reasons, such as:

• Racing Writes / Concurrent Updates: Multiple writes occurring simultaneously can result in a stale cache.
• Missed Writes / Failing to Update the Cache: Failure to update or set cache correctly after a database write.
• Delayed Cache Updates or Deletes: Slow propagation of updates or invalidation can leave the cache out of sync.
• Application-Level Bugs: Bugs in application side caching logic.

Maintaining cache consistency with the database is crucial for data accuracy, especially in distributed systems with concurrent web requests. Consistent caching ensures reliable read-after-write behavior, improving performance and user experience. Without it, applications may face unpredictable behavior and user frustration. A well-designed caching system boosts performance, ensures consistency, and delivers up-to-date data even under high concurrency.

Racing Writes

Let’s look at the scenario where two writers, A and B, update the same user in the database but write to the cache in a different order, causing the cache to become inconsistent with the database.

Author’s Note: In examples moving forward we’ll use “User_1” to mean id=1 in the “users” table.

Sequence of operations:

1. Writer A updates the name of user_1 to Foo in the database.
2. Writer B updates the name of the same user_1 to Bar in the database.
3. Writer B updates the cache for user_1 with name = Bar.
4. Writer A updates the cache for user_1 with name = Foo.

As shown, the cache and database end up out of sync because the cache updates by Writer A and B occurred in a different order than the database updates. This results in an inconsistent name value for user_1 between the database (Bar) and the cache (Foo). Essentially, this means stale writes were allowed in the cache.

From the above example, it’s clear that the cache allows stale copies to overwrite newer updates, leading to inconsistencies. Since execution order in distributed systems is beyond our control, our objective is to equip the cache with mechanisms to detect and reject stale updates to prevent such discrepancies.

Our Approach

To ensure the cache can distinguish between new changes and stale ones, we need a mechanism to identify and reject stale updates while correctly applying newer ones. This can be achieved by introducing a version identifier for database table rows and using this version identifier to reject stale cache updates.

Adding row versions to the database

While there are different approaches to generating row version information for database rows, the key part is that the client must not participate in the process. That is, the next version value must be generated on the database side in such a way that it is unique and monotonic.

Author’s Note: Simply using a timestamp is not sufficient because it is not unique and is not always monotonic, as one might hope it would be.

Our solution was to introduce a new column called db_version to our tables. To handle the incremental versioning we implemented a Postgres database trigger to handle this. This trigger increments the version for each update and initializes it to 1 for the first insert.

Postgres Database Trigger:

CREATE OR REPLACE FUNCTION uvf_update_db_version()
                RETURNS TRIGGER AS $$
                BEGIN
                    IF TG_OP = 'INSERT' THEN
                        NEW.db_version = 1;
                    ELSEIF TG_OP = 'UPDATE' THEN
                        NEW.db_version = OLD.db_version + 1;
                    END IF;
                    RETURN NEW;
                END;
                $$ LANGUAGE plpgsql;
CREATE TRIGGER uv_update_db_version BEFORE INSERT OR UPDATE ON <table_name> FOR EACH ROW EXECUTE FUNCTION uvf_update_db_version();

Author’s Note: If you plan to use a trigger or UDF, ensure that your database provides the same consistency level as your update. That is, you want the row update and version update to be an atomic operation from a consistency perspective.

When our application updates a row in the database, it also needs to update the cache with the new value. Since Django’s ORM does not return the updated row, we perform a select query after the update operation to retrieve the incremented version from the database. Both the update and select queries are executed within a transaction block.

Adding Version to the Cache

Since we added version information to database tables, we also need to include versioning in the cache to detect and reject stale writes.

To achieve this, we incorporate the version as a metadata header in the cache value. This header contains the version information and additional metadata. For more details, refer to Part 3: Appropriately serializing data for caching.

The version is included as a metadata header to maintain separation between serialization methods and cache functionality. This means the cache doesn’t need to know how to interpret the payload to get the version information from the serialized object.

Atomic Cache Operations

To correctly update the cache, we need to compare the supplied and stored version and perform the update as an atomic operation. Doing this on the client side is not feasible because we operate in a distributed and concurrent environment, where multiple cache updates to the same data can occur simultaneously. If the version check happens on the client side, there is a risk of stale writes, as the version may have changed after the check but before cache update.

A similar issue can exist on the cache side if the compare and update are not an atomic operation. We were able to leverage Redis’s single threaded nature to our advantage because it naturally protects against concurrent access. This allowed us to write custom Lua functions, executed by Redis, to perform conditional updates and deletes without worrying about concurrency issues.

Custom Lua functions

To effectively detect and reject stale writes to the cache, we implemented two Lua functions to perform conditional updates and deletes. The function stubs are:

--- Sets a key only if the provided version is greater than the stored version or if the key does not exist.
-- If the supplied version is equal to or lower than the stored version, the operation is rejected.
-- @param key (string) Name of the key.
-- @param version (string) Corresponds to the database version value.
-- @param metadata (string) Additional header information to store.
-- @param value (string) Data to associate with the key.
-- @param ttl (number) Time-to-live for the key in seconds.
-- @return (string) "OLD" if the supplied version is lower than the stored version.
--                  "OK" if the key was set or if versions match.
function set_if_version(key, version, metadata, value, ttl)
   ...
end

--- Deletes a key only if the stored version is less than the supplied version.
-- @param key (string) Name of the key.
-- @param version (string) Corresponds to the database version value.
-- @return (number) 0 if the key was not deleted or did not exist.
--                 1 if the key was deleted.
--                -1 if the key was not deleted because the stored version is greater than the supplied version.
function del_if_version(key, version)
   ...
end

Racing Write Example

Let’s bring everything together and see how we handle the racing write problem and prevent stale writes to cache.

Sequence of operations:

1. Writer A updates the name of User_1 to Foo in the database.
2. Writer A receives db_version=12 from database
3. Writer B updates the name of the same User_1 to Bar in the database
4. Writer B receives db_version=13 from database
5. Writer B updates the cache for User_1 with version=13 using set_if_version
6. Writer B receives OK from set_if_version call since it updated the cache
7. Writer A updates the cache for User_1 with version=12 using set_if_version
8. Writer A receives OLD from set_if_version call since it supplied old version and the cache update was rejected

With this solution, we can effectively detect and prevent stale writes to the cache.

Conclusion

To address cache inconsistencies due to racing writers, a look-aside cache must have a way to reject stale writes. This will mean each database row version will need a unique and monotonic version and the cache must perform conditional updates in a serializable manner.

This approach is a building block to solve more advanced consistency edge cases such as missed writes and racing cache fills. To learn more about these issues and the solution we implemented, stay tuned for Part 5: A Time-Bounded Eventually-Consistent Cache of the Scaling Nextdoor’s Datastores blog series.

Authors: Ronak Shah, Slava Markeyev, and Tushar Singla

Scaling Nextdoor’s Datastores: Part 4 was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In part 1 of this series we discussed how ORMs, object-relational mapping frameworks, help abstract away database specific schemas and queries from application code. Developers simply utilize objects in their application’s language to access database data.

Here’s a simple example of using Python’s Django ORM to define a model:

from django.db import models

class Users(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

The associated SQL create table would look like:

CREATE TABLE users (
    "id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
    "first_name" varchar(30) NOT NULL,
    "last_name" varchar(30) NOT NULL
);

Developers would then access database data like this:

user_id = 123
user = User.objects.get(id=user_id)
print(user.first_name)

Object Byte Serialization for Caching

An issue arises when adding a look-aside cache such as Redis/Valkey to an application: How do you store what you got from the database in the cache?

A common solution to caching complex objects, such as those from ORMs, is object byte serialization. This process converts language objects into bytes before storing them in the cache. When reading from the cache the process is done in reverse where the byte data is turned into language objects. For instance in Python this is often done with the pickle package.

The interaction between the application, database, and the cache looks like this:

import pickle

# Try getting from cache ('None' if not in cache)
user_bytes = cache.get("user_123")

if user_bytes is not None:
   # Read bytes using pickle
   user = pickle.loads(user_bytes)
else:
   # Fetch from database
   user = User.objects.get(id=123)
   
   # Convert to bytes
   user_bytes = pickle.dumps(user)

   # Store in cache
   cache.set("user_123", user_bytes)

While this method enhances performance by reducing database queries and leveraging cached data, it also presents challenges, particularly when serialized data is tightly coupled to specific runtime environments, package versions, and schema definitions.

Issues with Byte Serialization

Bound to Runtime Version and Package Version: Serialized objects often embed information about the object’s structure as defined by the code at the serialization time. When the code or its dependencies are updated (e.g., a new release), serialized objects may fail to be deserialized, making the cache data incompatible with the new code version.

Thundering Herd Problem during Migrations: When a schema migration occurs (e.g. adding a new field/column), the cache might suddenly contain a mix of old and new serialized data. As a result it can force the application to treat many cache entries as misses due to deserialization failures. If the cached items are large or in high demand (“hot”), a simultaneous cache miss across many clients can result in a “thundering herd” effect. In this scenario, numerous processes will concurrently query the database to refill the cache, placing excessive load on the database.

Author’s Note: The implied solution to deserialization errors is to query the database and re-fill the cache.

Forward and Backward Compatibility

To address the challenges discussed above, it is important to design a cache serialization strategy that ensures both forward and backward compatibility.

• Forward Compatibility: Older versions of the application should be able to read cache entries that were written by newer application versions.
• Backward Compatibility: New versions of the application should be capable of reading cache entries that were written by previous application versions.

Serialization format of choice

When evaluating our options, we compared various serialization formats based on versioning compatibility, the performance of serialization and deserialization, and the resulting serialized byte size. Ultimately, we chose MessagePack to serialize Django Model objects.

Forward and Backward Compatibility

For forward compatibility, the MessagePack Python library can ignore new fields during deserialization if they don’t exist in the current version of the code. This ensures that older versions of the code can still deserialize cache entries from an updated data model.

Author’s Note: Since we rarely remove fields from Django Models, older code typically doesn’t encounter missing fields in the cache. When field removal is necessary, we use Django’s SeparateDatabaseAndState to decouple database schema changes from model updates.

For backward compatibility, when the new code version deserializes old cache entries, it populates any missing fields from the cache with their default values. We require developers to provide default values when adding new fields.

Performance

To evaluate MessagePack’s performance, we conducted extensive tests comparing its standalone serialization/deserialization performance and its performance when integrated with our application. Our findings showed that MessagePack is slower than some alternatives (e.g., pickle). However, in our application tests, this difference did not noticeably affect overall latency when handling web requests.

Serialization Byte Size

MessagePack produces a smaller byte stream compared to other formats. Additionally, when combined with compression methods, we can further reduce the size of the serialized data.

Implementation Details

To support additional use cases and potential future revisions to cache storage, such as compression, we prepend additional information to the serialized value prior to writing it to the cache.

The prepended header is composed of two parts: a 2-byte metadata field and an 8-byte (64 bit) version field. For example, a serialized value might look like this:

\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01<messagepack bytes>

• The first 2 bytes (\x00\x00) represent the metadata where we store serialization format information.
• The following 8 bytes (\x00\x00\x00\x00\x00\x00\x00\x01) encode object version information.
• The remaining bytes are the MessagePack-serialized data.

We’ll explore the role and necessity of the version information bytes, and how we use them in Part 4: Keeping the Cache Consistent.

Conclusion

Ensuring forward and backward compatibility in caching alone wasn’t enough to deliver accurate data to users — we also needed a way to maintain cache consistency with our database. Maintaining cache consistency is crucial in a distributed environment, where we are handling concurrent web requests while delivering up-to-date data to users. To learn more about the importance of cache consistency and how we achieve it, check out Part 4: Keeping the Cache Consistent in the Scaling Nextdoor’s Datastores blog series.

Authors: Ronak Shah, Slava Markeyev, and Tushar Singla

Scaling Nextdoor’s Datastores: Part 3 was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Adding read replicas to an existing database is a very common pattern as applications or products evolve to handle increased demand. Typically, the implementation details are hand waved and it’s assumed that this strategy will work. However, that is rarely the case, and we’ll dive into some more of the intricacies around the implementation.

Initial Attempt

When replicas were first introduced in the Nextdoor stack, we gave the product engineers latitude to choose when they wanted to have their query routed to a read replica or to the primary. This was done by leveraging the existing routing mechanism in our ORM, Django.

This seemed like the right idea at the time because the product engineers had the most context around consistency requirements within their changes and load characteristics of their product feature. Therefore, they would have the best ability to judge which node to send their query to. However, as our business logic evolved and became more feature-rich, product engineers began to add abstraction layers to help abstract complex operations away from business logic.

The explicit routing decisions engineers made became buried and subsequently created a serious problem for users of these abstractions. If one abstraction method was performing a write and another a read, they could not safely be used together due to read-after-write consistency issues. Due to replication lag between the primary and replica databases, a race condition arises when the application attempts to read data from a replica after performing a write.

We had created a system where engineers had to be aware of the entire call stack and be able to determine if this situation would apply to them or not. The easiest way to handle this situation? Always use the primary…

The Band-aid

A common solution engineers employed was to wrap higher-order business logic in database transactions because within the context of a transaction, all queries are routed to the primary.

Author’s Note: Our default isolation level for transactions, repeatable read, gave engineers a false sense of security, as it only guarded against replication lag and not racing writes with concurrent transactions. We have since improved this to ensure read-your-own-write semantics.

This strategy had a negative effect on database load because it indiscriminately caused all queries intended to be sent to replica databases to be sent to the primary database. The impact of this problem increased as:

1) more business logic leveraged the database

2) business logic increased its use of existing abstractions

3) query performance decreased as more data was added

What transpired was a years-long erosion of the capacity benefits the additional read replicas initially provided. As a result, the load on the primary database node became one of the most pressing issues with Nextdoor’s database stacks.

It was clear that the initial approach of exposing routing choices to engineers was no longer tenable and the team embarked on a way of making the replica vs. primary decision for the product engineers.

Reimagining

Using ORMs (Object Relation Mappings) is controversial. There are many pros and cons and we won’t debate all of them here. However, one of the advantages of an ORM is that there is a consistent layer of abstraction between the database and the application. This allowed us to inject a simple piece of custom logic to keep track of which tables have been written to while processing a web request. Why is it helpful to keep track of modified tables? By doing this we could automatically make informed decisions of where to route subsequent read queries, regardless of where they were buried in the business logic stack.

This simple strategy, coupled with our already high read-to-write ratio, allowed us to shift much of the read traffic to the replica databases and substantially reduce our reliance on the primary database.

Author’s Note: While the strategy was simple, we did have to cleanup up all of the manual routing decisions along with inappropriate usages of transactions.

While this naive approach was rather effective, we realized that this strategy was actually too conservative. We noticed that our average database replication lag was around 20ms while our web requests lasted an order of magnitude longer. That means that even after the update had been replicated to the primary, we were still disallowing queries for that table to the read replica. This provided an opportunity to use a timing based system that marked tables as re-eligible after p99.9 replication lag had elapsed. With this additional optimization, we were able to re-route most of the queries from our primary to the read replicas.

Takeaway

Each application will have its own eccentricities that make RDBMS scalability a challenge for platform teams but we hope this post provides a cautionary tale, as well as a potential solution for similar cases.

In the next post, Appropriately serializing data for caching, we’ll cover how we serialize database data for caching and some of the pitfalls to be aware of when introducing caching to an application.

Authors: Tushar Singla, Slava Markeyev and Ronak Shah

Scaling Nextdoor’s Datastores: Part 2 was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. How can we reduce load on our primary database(s) and better utilize database read replicas?
2. How can we improve our cache consistency?

In this post we’ll provide a primer on the common industry-wide solutions we’ve previously employed along with discussing their caveats and pitfalls. In subsequent posts we’ll dive into the technical details of the components of our solution and how they fit together.

Table of Contents

1. Background primer (this post)
2. Decreasing database load with dynamic routing
3. Appropriately serializing data for caching
4. Keeping the cache consistent
5. A time-bounded, eventually-consistent cache

Background

Nextdoor’s backend, built using the Python-based Django web framework, powers the core product experience for neighbors, government agencies, and local businesses. The power of Django and similar frameworks (Rails, Spring, etc) is that they allow development teams to focus on implementing business logic rather than getting caught up in the details like learning and writing SQL.

The Object Relational Mapping, ORMs, included in these frameworks provide a lever that allows developers to define data models and relationships between them in the application’s language without ever needing to worry about SQL.

As some readers are all too aware, relational data modeling comes at a cost. Without careful data modeling, performant access to relational data largely depends on that data residing on monolithic databases.

NoSQL or distributed SQL datastores are often advertised as solutions to the scalability challenges of relational databases like PostgreSQL. However, many companies face significant obstacles in transitioning to these modern datastores. Their relational data models are deeply entrenched, and much of their business logic relies heavily on the power of relational data.

This was the position the Core-Services team at Nextdoor found ourselves in at the end of 2023. We had explored changing our data model to one that could properly leverage distributed SQL but we ran into existential challenges supporting existing relational queries, specifically multi JOINs, that were scattered across our codebase. Our small team simply could not rewrite business logic to avoid these queries and we couldn’t stop all product development to have other teams do this either.

Common solutions

Before diving into our most recent solution, it’s worthwhile to discuss how we got to this point. Many platform teams often begin with a basic architecture of the application, via an ORM, talking directly to a relational database like PostgreSQL.

Caching

A ubiquitous pattern that arises in the backend infrastructure lifecycle is adding a cache to take load off of the database. A popular solution, one that Nextdoor employed, was adding a look-aside cache powered by Redis.

When using a cache, a few common points of consideration arise:

• You likely want to maintain the flexibility the ORM provides with the benefits of caching. Due to the ORM’s flexibility in querying complex relations, not all queries will be cache-able. What then?
• If performing a read with the intention of doing a write, engineers probably don’t want stale data from the cache. An escape hatch must be built in.
• How do product engineers interact with the cache and database?
• What is actually stored in the cache and how is it formatted? Will the application still be able to read what’s in the cache if the data model / schema changes?
• If the data in the database changes, how does the cache get updated?

To be clear, not all of these points need to be addressed when initially deploying a cache but they do creep up over time. Like many companies, Nextdoor solved these problems as they started to appear on the horizon. Later parts of this blog series cover components that solve many of these problems, so stay tuned!

Read Replicas

Another common infrastructure improvement is adding database read replicas to serve queries that can’t be answered by the cache.

This approach helps address database load on the primary but does not entirely solve the problem. Common issues that arise:

• Engineers may not be able to reason about the various consistency pitfalls that come with this architecture, nor should they if their goal is to simply build product features. If an option is provided, they will choose the perceived safest one: sending their queries to the primary database.
• Business logic will often need to perform some reads followed by a write. This will typically be wrapped in a transaction which will mean this must run on the primary database.
• In order to avoid consistency problems, populating the cache on a cache-miss will likely need to be done by querying the primary database. However, even if the primary database is queried, it does not guarantee that what gets written into the cache is the most up to date version of the data.

Data Partitioning

Another common solution which teams will employ is splitting up their data across multiple physical databases. This strategy has several different approaches such as splitting by tenancy or breaking up foreign-key relations. Due to our existing data model and access patterns we chose the latter option of severing foreign-key relations such that a set of tables could be moved into their own physical database. While our ORM handled routing to the appropriate database depending on what was being accessed, it did require us to carefully rewrite some business logic.

This comes along two key caveats:

• This strategy is valid and may be the only option but it only prolongs the runways and does not solve the fundamental problem that a physical database can only be so big.
• Transactions which exist in business logic may now be subtly broken in that they no longer provide atomicity when dealing with successive writes to different databases.

Teams like ours may eventually find themselves back at square one where the primary database(s) are still a bottleneck and single point of failures despite the efforts to mitigate these problems. Through thoughtful data modeling and incremental work, the problem of database scalability doesn’t have to be immediately existential to businesses. Despite all of the caveats, the above strategies will add years of runways before the problem is well and truly existential.

Next Evolution

When reevaluating our architecture and system behaviors, we aimed to alleviate the database load issue while also addressing complementary problems discussed in the caveats listed above. Specifically we wanted to:

1. Automatically route database queries to read replicas whenever possible.
2. Perform cache filling from the database read replicas.
3. Provide guaranteed eventual consistency of the cache in a timely manner.
4. Allow the application to be able to use what was in the cache even if a developer added a new field/column to a data model.

Check out the next post in this series on how we employed dynamic routing to shift load from our primary database to the read replicas.

Authors: Slava Markeyev, Tushar Singla, and Ronak Shah

Scaling Nextdoor’s Datastores: Part 1 was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Nextdoor is the neighborhood network where neighbors, businesses, and public agencies connect with each other. Nextdoor is building innovative solutions to enhance the user engagement with AI-Generated Content (AIGC). This post outlines our approach to improving user engagement through user feedback, specifically focusing on Notification email subject lines. Our solutions employ Rejection sampling [1], a technique used in reinforcement learning, to boost the engagement metrics. We believe our work presents a general framework to drive user engagement with AIGC, particularly when off-the-shelf Generative AI falls short in producing engaging content. To the best of our knowledge, this marks an early milestone in the industry’s successful use of AIGC to enhance user engagement.

Introduction

At Nextdoor, one of the ways to drive user growth and engagement on platform is through emails. One of the emails we have is called New and Trending notifications, where we send a single post that we think the user might be interested in and want to engage with. As part of sending an email, we need to determine a subject line of the email for the email audiences. Historically, we simply pick the first few words of the post being sent to be the subject line. However, in certain posts, these initial words are often greetings or introductory remarks and may not provide valuable information to the user. In the provided image example below, we observe a simple greeting, “Hello!”

In this work, we aim to use Generative AI technologies to improve the subject line. With Generative AI, we aim to generate informative and interesting subject lines that will lead to more email opens, clicks and eventually more sessions.

Writing a good subject line with Generative AI is challenging because the subject line needs to satisfy the following criteria. First and foremost, the subject line needs to be engaging so that the users want to open the email. To see if ChatGPT API can write engaging subject lines, we tried generating subject lines with ChatGPT API with a small traffic A/B test, and found that the users are less likely to click on emails if we use subject lines made by ChatGPT API (e.g. Table 1). As we show later, we tried to improve the prompts (prompt engineering) but the results were still inferior to the user-generated subjects. This finding implies that Generative AI models are not trained to write the content that is particularly engaging to our users, and we need to guide Generative AI models to increase user engagement.

Second challenge is that the subject line needs to be authentic. If it reads like a marketing phrase, the email will look like spam. The example in Table 1 “Support backyard chickens in Papillion, NE!” shows this issue.

Third, the subject line should not contain hallucinations (a response that is nonsensical or not accurate). And it is well known that Generative AI is vulnerable to hallucinations [2]. For example, given a very short post saying “Sun bathing ☀️”, ChatGPT API in Table 1 generated the subject line “Soak Up the Sun: Tips for Relaxing Sun Bathing Sessions”, which had nothing to do with the post content.

We developed a novel Generative AI method to overcome the three challenges faced by the ChatGPT API mentioned above. We made three contributions:

• Prompt engineering to generate authentic subject lines with no hallucination: Given a post, ChatGPT API creates a subject line by extracting the most interesting phrases of the post without any rewriting. By extracting the user’s original writing, we are able to prevent marketing phrases and hallucinations.
• Rejection sampling with a reward model: To find the most interesting subject line, we develop a reward model whose job is to predict if the users would prefer a given subject line over other subject lines. After ChatGPT API writes a subject line, we evaluate it by the reward model and accept it only if its reward model score is higher than the user-written subject line’s score. This technique is called Rejection Sampling and recently introduced to Reinforcement Learning for Large Language Model training [1].
• Cost optimization and model accuracy maintenance: We added engineering components to minimize the serving cost and stabilize the model performance. By using caching, we reduced our cost to 1/600 compared to the brute-force way. By daily performance monitoring, we can catch if reward models fail to predict which subject is more engaging due to external factors such as user preference drift and address it by retraining.

We believe that this framework is generally applicable when off-the-shelf Generative AI fails to improve user engagement. We also analyzed the importance of each component in our design. Even with the aforementioned prompt engineering, ChatGPT API did not necessarily produce more engaging content. This highlights the necessity of the rejection sampling component: in such cases, we can develop another AI model as a reward model and use the Generative AI’s output only if the reward model approves [1].

Proposed Method

For every post, we employ the following system to create a subject line. It’s important to mention that we generate a single subject line for each post, without personalization. This decision was made to minimize computational cost. Exploring cost-effective methods for implementing personalized subject lines will be an interesting future work.

Model Overview

Figure 2 illustrates our approach. We develop two different AI models.

• Subject line generator: This model generates a subject line given a post content.
• Reward model (Evaluator): Given a subject line and the post content, this model predicts if the given subject line would be the better subject line than the user-generated subject line.

Given a post, the Subject line generator produces subjects in Figure 2 (green boxes). The reward model compares the OpenAI API subject line (green) with the user-generated subject line (red), and selects the more engaging one. For the top post, the OpenAI API subject line contains more relevant information and is selected. For the bottom post which was about a health alert, the reward model selects the user-generated subject. While the OpenAI API subject line shows the main content of the alert, the reward model picks the user-generated subject because it shows the importance of the post and thus is more engaging.

Developing Subject Line Generator

We use OpenAI API without fine-tuning. In the prompt, we require that OpenAI API extracts the most interesting part of the post without making any change. This way of extracting user content provides multiple benefits: First, it removes hallucinations. Second, it keeps the subject line authentic as OpenAI API does not rewrite the original content. To test the prompt engineering, we A/B tested generator outputs without reward models. We found that asking OpenAI API to extract in the prompt improves Sessions by 3% relatively compared to asking OpenAI API to rewrite the subject line from scratch (See the Results section for the details).

Developing Reward Model

We fine-tune OpenAI API to develop a reward model. This is the main innovation we applied on top.

Training data collection: The challenge is to collect training data on which subject line was more engaging. Manual annotation is not possible because there are no rules deciding what subject line is more engaging. We found that the subject lines that we thought to be more engaging than the user-generated ones turned out to be less engaging (Table 2).

To tackle this issue, we collect training data via experimentation. For each post, we generate subject lines in two ways. One way is to use user-generated ones and the other is to use the OpenAI API generator described above. Then we serve 2–3% users (~20k) that are randomly selected with each subject line. The goal is to learn which subject line was more engaging through click data.

Model training: We used OpenAI API to fine-tune with the labels we collected. We used ~50k examples and 40% of examples had the OpenAI API subject as the winning subject and the rest had the user subject as the winner. Given a subject line and post content, our model is fine-tuned to predict if the subject line would generate more engagement (clicks) than the user-generated subject line. The model is asked to predict if the subject line is more engaging and output “Yes” or “No”.

Training details: We used the smallest OpenAI API model “ada” for fine-tuning. We found that larger models did not improve the predictive performance despite higher cost. We added a logit bias of 100 for “Yes” and “No”. These biases boost the probability for the model to output “Yes” or “No”. We tried to change the number of epochs and selected the model with 4 epochs, but we did not see much difference in offline performance after 2–3 epochs.

Engineering details: We added the following components for optimization and safeguarding.

• Caching: For each post, we cache the outputs of our model. By processing each post only once, we reduced the cost to 1/600. In other words, each post gets sent 600 times on average and we process the post only once instead of 600 times. Caching also optimizes the OpenAI API usages (the number of tokens and the number of requests).
• Reward model performance maintenance: We monitor the reward model’s predictive performance daily, using the next day’s user clicks after the training phase as the ground truth to compare with the model’s output. Model’s predictive performance can change because our users’ preference may change and the content in Nextdoor can shift in the writing styles or topics.
  For monitoring purposes, we collect the engagement performance of different subject lines in the following way. We created a “control” user bucket where we always send emails with the user-generated subject and a “always OpenAI API” bucket where we always send with the OpenAI API subject, regardless of the reward model’s output. From these two buckets, we know the ground-truth on which subject line was more engaging, and measure the reward model’s accuracy. If the accuracy goes down by 10+%, we retrain the reward model with new data.
• Retries with Fallback: Since OpenAI API may return an error due to the rate limit or transient issues, we added retries with exponential backoffs with Tenacity. If we fail after a certain number of retries, we fallback to the user-generated subject.
• Controlling the length of output: We found that the Subject line generator would write a subject line longer than our desired length (10 words). This happened even if we specified the 10 word limit in the instruction and added examples. We post-processed the generator output by cutting the first 10 words from the generator’s output. We A/B tested different word limits and found that 10 is the optimal value.

Results

We did A/B tests with different versions of the subject line generator, and with and without the reward model. For the generator, we tested the following options

• Writing with OpenAI API: We ask OpenAI API to “write an engaging subject line for a given post”. This was the first version we tested without much prompt engineering.
• Extracting with OpenAI API: We ask OpenAI API to extract the most interesting part and provide 5 examples. We also add requirements in a numbered list such as “Do not insert or remove any word.”, “Do not change capitalization”, “If the first 10 words are interesting, use them as a subject line”. We tried 4 different versions of prompts and picked the best version by A/B test metrics.

For the A/B test metrics, we primarily focus on Sessions. A session is an activity sequence made by the same user within a certain timeframe, and sessions quantify the number of unique user visits.

Table 3 shows the results on Session lift compared to the “control” bucket where we use user-generated subject lines. In addition to the session metrics, our final model (last row) increased Weekly Active Users by 0.4% and Ads revenue by 1%.

Here is what we learned from A/B tests:

• Prompt engineering improves the performance but has a ceiling. After a few iterations, the A/B test metrics showed only marginal improvements, failing to beat the control.
• Finding the “optimal” prompt is an elusive task, as the space of potential prompts is boundless, making it difficult to explore. Moreover, there is no established algorithmic or systematic method for enhancing prompts. Instead, the task relies on human judgment and intuition to update the prompt.
• Reward model was the key factor in improving sessions.
• Predicting popular content is challenging, as is the reward model’s task of forecasting popular subject lines, which currently achieves about 65% accuracy. Enhancing the reward model’s performance by leveraging real-time signals like the current engagement numbers for the subject can be an interesting future work.

Conclusions

We developed a novel Generative AI system to increase user engagement by combining the reward model and prompt engineering. Our systems have engineering components for cost saving and monitoring. A/B tests showed that our systems can deliver more engaging subject lines than the user-generated subject lines.

There are many avenues for future work. First is to fine-tune the subject line generator. In this work, we used vanilla ChatGPT API as the generator. Instead, we can fine tune OpenAI API with the most engaging titles that the reward model identifies. For each post, we generate multiple subject lines and use the reward model to pick the winner. Then we use the winner subject to fine tune the subject line generator. This approach is called Reinforcement Learning by Rejection Sampling [1].

Second is to rescore the same post daily. Currently, we pick the best subject line with a reward model once and never rescore. However, as time goes on, we may be able to see which of the OpenAI API subject line or user-generated subject line is getting more engagement, and our reward model can predict more accurately. Third is to add personalization without significantly escalating computational costs.

Acknowledgments

The post was written by Jaewon Yang and Qi He.

This work was led by the Generative AI team with cross-org collaboration between Notification team and ML teams. We would like to give a shout out to all the contributors:

Jingying Zeng, Waleed Malik, Xiao Yan, Hao-Ming Fu, Carolyn Tran, Sameer Suresh, Anna Goncharova, Richard Huang, Jaewon Yang, Qi He

Please reach out to us if you are interested to learn more — we are hiring!

References

[1] Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models, Arxiv preprint, 2023

[2] Ji et al. Survey of Hallucination in Natural Language Generation, ACM Computing Surveys, 2022

Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

The majority of ML models at Nextdoor are typically driven by a large number of features that are primarily either continuous or discrete in nature. The personalized features usually stem from historical aggregations or real-time summarization of interaction features, typically captured through logged tracking events. However, representing content through deep understanding using information behind it (text/image) is crucial for modeling nuanced user signals and better personalizing complex user behavior across many of our products. In the rapidly evolving field of NLP, utilizing transformer models to perform representation learning effectively and efficiently has become increasingly important for user understanding and improving their product experience.

Towards that, we have built a lot of entity embedding models spanning entities such as posts, comments, users, search queries & classifieds. We first leveraged deep understanding of content and used that to derive embeddings for meta entities like users based on their past interacted content. These powerful representations are found to be very crucial towards extracting meaningful features for some of the biggest ML ranking systems at Nextdoor such as notifications scoring and feed ranking. By making them readily available and building to scale, we can drive adoption of state-of-the-art reliably and put them in the hands of ML Engineers for rapidly building performant models across the company.

This blog primarily focuses on how we iterated on the development of embedding models, how they are featurized and served at large scale into various product applications as well as some of the challenges encountered during this process. We summarize the evolution of work across three sections. In section 1, the focus is to leverage state-of-the-art pre-trained models to rapidly evaluate the value of embeddings models as feature extractors. Section 2 describes how to fine-tune embeddings using unlabelled data for certain products, whereas Section 3 demonstrates the use of labeled data to fine-tune embeddings for better task prediction. This work is driven by the Knowledge Graph Team at Nextdoor, a horizontal team that works in close collaboration with product ML teams as well as the ML Platform team who owns the ML training and serving platform and the FeatureStore service powering ML models at Nextdoor.

1. Leveraging Pre-trained models

The first generation of embeddings are built from pre-trained language models using the Sentence-BERT paradigm (https://www.sbert.net/). SBERT is well-known to produce better embedding representations compared to original BERT models [1]. The main goal here is to rapidly experiment with embeddings as features and realize their value in the product as quickly as possible. The text from content entities, viz. Nextdoor posts & comments, is extracted from post’s subject and body and comment text respectively, which is then fed into a multilingual text embedding model to derive respective entity embeddings for all countries Nextdoor operates in. For a given user, their historical interacted posts’ embeddings are weighted aggregated based on interaction type to inform user (interaction) embedding. Ex: Active interaction such as post creation/comment/click would have higher weight compared to a more passive interaction like impression. These signals are aggregated across both online (feed) and offline (emails) product surfaces to represent user embedding holistically and are updated daily for all users in the platform.

These features were found to be among the most important features for multiple ranking models and delivered significant performance lifts in key product OKR metrics across both notifications and feed when shipped in early 2022. The pre-trained models also served as a good proof-of-concept to build out reliable feature ingestion pipelines and monitoring systems identifying any potential feature drifts and disruptions. This helped form a robust playbook for deploying several next generation embedding features.

2. Fine-tuned embeddings from unlabeled data

The next generation of embeddings describes training custom models which are improvements over pre-trained versions by leveraging techniques of fine-tuning. The signals used to generate embeddings earlier come from user interactions across notifications and home feed products either directly or indirectly. In contrast, this section details a use case that makes use of unlabelled data to perform representation learning to improve user search experience.

Our neighbors use Nextdoor search to find useful local information by expressing intent explicitly. We tried to capture both long and short term intent to determine and serve user perennial (e.g.: home maintenance) as well as ephemeral needs (e.g.: lost & found). Search queries — while being high intent in nature, are inherently short and noisy. A searcher might try multiple variations of a query successively in order to get their intent fulfilled as much as possible. Additionally, due to the nature of local search, relying on labeled feedback from search results may not fully capture user intent due to limited liquidity.

To fully capture user intent signals, we rely on a self-supervised training strategy to learn fine-tuned representations for any given query. Specifically, we first built an SBERT backed query embedding model that learns to embed search queries in lower dimensional space. Then, we aggregate embeddings from user queries across different time windows (weekly/monthly/quarterly periods) to generate multiple user (intent) embeddings. The same model also extracts the intent of a post to generate the corresponding post embeddings. The resulting user, post and query embeddings are transformed and featurized as described in the later section to improve the performance of the ranking models.

The query embedding model is originally built to drive contextual query expansion in Nextdoor search pipeline [2]. This sentence transformer model is trained on historical search queries in order to best learn query representations. We first collected search logs that consisted of sequences of search queries within a session across all searchers over a period of time. Then, they are pre-processed using traditional NLP methods like lemmatization, spell checking, deduplication etc. to form a clean corpus of tokens, which is composed of n-grams (n=1,2,3) and whole queries. To generate a training dataset, we created positive pairs of tokens occurring within a user search session and negative pairs randomly occurring across sessions. Contrastive learning with cosine similarity loss is used to train the underlying model.

For the query expansion use-case, this model drove better contextual search results by identifying related candidates improving recall. This helped not only improve key search metrics across content search and product search in For Sale & Free but also reduced the rate of null queries significantly compared to prior word embedding models. We also leveraged HSNWlib [3], an approximate nearest neighbors library to implement this deep learning based query expansion further improving expansion latencies by more than 10x. For notification & feed use cases, intent features generated from transformations of post & user embeddings helped achieve significant positive impact on our top line engagement metrics. Although features can only be computed for searchers and are of low coverage overall, this explicit signal is found to be very useful in improving the overall search experience.

3. Fine-tuned embeddings from labeled feedback

In the next evolution of embeddings, we additionally leverage user feedback to fine-tune models further. The pre-trained entity embeddings have served us well over a year, but they are off-the-shelf models trained using public benchmark datasets. As such, their semantics are quite different in nature from the Nextdoor domain. Moreover, their high but fixed model dimensionality contributes to significant storage and serving costs, especially when user embeddings are updated for all Nextdoor neighbors daily. To address these, we built a two-tower framework to fine-tune embeddings with user feedback collected across Nextdoor surfaces while reducing dimensionality, customizing to our domain, and being cost effective.

The fine-tuned models are developed and trained in phases, incrementally adding complexity. In the first phase, the inputs to post and user towers are pre-trained embeddings, which are then transformed using multiple FC layers, reducing dimensionality at each step. The standard cross-entropy is used as a loss function to predict the task of notification clicks for a given user and post. To generate a training dataset, we sampled from random explore logs to reduce selection bias, the same process as that of the downstream ranking model. Once the model is fully trained, the last layer generates fine-tuned user and post representations.

These pytorch models are trained on millions of records using SageMaker GPU instances with varying hyperparameters, and the model with the best offline performance is chosen to generate & store fine-tuned embeddings into FeatureStore. The earlier described playbook is followed to build and monitor offline and online feature pipelines. Serving these cached features to downstream models has shown promising lifts in all engagement metrics (CTR/sessions/contributions/DAU/WAU) while keeping guardrail metrics that measure harmful/hurtful content distribution across the platform neutral.

In the next phase, we fed the post tower directly with text extracted from the post entity, allowing us to fine-tune parameters of the SBERT model. The test AUC score is used as a benchmark to determine how many layers and transformer blocks to unfreeze for trying out different training schemes along with optimization of hyperparameters of typical DNN models. The best model also improved the user — post cosine similarity of fine-tuned embeddings by up to 16% when compared to respective pre-trained versions — an additional evaluation criteria of intrinsic quality of improvement in representations. It is also noteworthy that this quality improvement is achieved while reducing dimensionality by more than 10x!

In the most recent phase, we extended into multi-task learning (MTL) setup modeling both notification clicks and feed actions to jointly optimize learning of fine-tuned embeddings. Again, these objectives mimic downstream rankers exactly to make sure learnt embeddings directly optimize downstream tasks. MTL models have the added advantage of learning a single model across multiple product surfaces thereby reducing operational burden and maintenance costs, while leveraging knowledge transfer across shared tasks for better representations. The feed and notifications surfaces are highly related as clicking on email notifications lands directly into the pinned view of the post in the newsfeed. Additionally, most actions on home feed are used as features in notification ranker making these tasks very related.

Using embeddings in ML models

As most of our downstream production models are tree based models, they don’t directly integrate with vector features like embeddings, like that of Deep neural networks. Therefore, we primarily use outputs from embedding models as feature extractors into downstream models. Specifically, we rely on transformations like cosine similarity & dot products across these entity embeddings in order to generate meaningful affinity features. While transitioning into neural network systems is currently underway — these vector transformations provide a neat way to integrate embedding based features into existing models and enable rapid experimentation for assessing performance lift of new deep features.

We first create schema and declare feature groups corresponding to each embedding to host within our in-house FeatureStore. Then, content based embedding features are ingested into our Featurestore in near real-time using task worker jobs as they get created/updated. For users, the daily scheduled jobs in Airflow compute embedding aggregations based on pre-specified lookback windows, weighted across various interaction types, and are batch ingested into Featurestore. Once systems are set up to ingest all relevant embeddings with appropriate TTL, we then write logging code to compute and log the derived features such as cosine similarity and dot product between user & post, user & user. Specifically, in feed, these features would represent affinities between post vs viewer, viewer vs author and analogously between post vs recipient, sender vs recipient in notifications world. Similarly, we also compute affinities across comment entities to inform activity based ranking in newsfeed. The data obtained from feature logging is used to train downstream ML ranking models, to avoid online-offline skew, and the model with best offline performance lift with new features is promoted for online AB test evaluation and ramping further towards majority member experience.

Challenges & Future

Multiple entity embeddings i.e user, post, comment, query etc have been successfully integrated into various product surfaces at Nextdoor at large scale. In the past, models based on comment embeddings helped foster and cultivate kinder conversations to improve platform vitality metrics [4]. More recently, contextual topic embeddings are also developed using BERTopic [5] to achieve coarser level personalization of content to neighbors, while informing us about prevalence of content categories and types across the platform. We are also experimenting with image embeddings using CLIP [6] to leverage image/video information behind content.

In addition, as an extension to labeled fine-tuning, we plan to further improve representations along two dimensions. One is by concatenating representations with additional features such as image embeddings and existing interaction features to leverage multimodal and dense signals. The other is to extend tasks to other surfaces such as ads, For Sale & Free (marketplace) etc to make representations more holistic across products. Once downstream models are fully modernized to DNN-based methods, the embeddings can be integrated into the model directly without losing any information from computation of transformations.

As we build more and more embeddings capturing different signals, we also need to be mindful of additional incurred costs from new features. Some of the initial challenges of serving high dimensional vectors during inference are mitigated by performing embedding transformations directly within FeatureStore rather than passing embeddings across microservices minimizing network bandwidth and scaling costs. This worked well with tree based models, however in the future, serving embeddings directly with DNN models can add up costs. Caching and serving fine-tuned embeddings can help control dimensionality while incorporating domain specific knowledge. This allowed us to rapidly experiment and quickly evaluate ROI at a smaller scale, justifying overall costs. From an infra standpoint, we found that optimizing the payload format of embedding features as well as sequencing of calls to efficiently read/write from FeatureStore greatly reduces overall costs at full scale.

Acknowledgments

This work would not have been possible without close cooperation and collaboration with various ML product partners (Notifications/Feed/Search/Vitality) as well as significant support for ML platform and FeatureStore service from ML Platform team. I would like to take this opportunity to give a huge shoutout to all the dedicated Nextdoor folks from these teams behind this endeavor.

Nextdoor is building the largest Local Knowledge Graph (LKG) in the world. The local knowledge graph inherited in our neighborhoods is Nextdoor’s unique proprietary data that can be used to enable personalized neighborhood and neighbor experiences. The Knowledge Graph team is focused on understanding neighbors and content by creating standardized neighbor/content data using state-of-the-art ML methods.

Third-party large language models (LLMs) such as GPT and the corresponding dialogue applications like ChatGPT, which are built upon these language models, lack access to the specific local knowledge of Nextdoor. As a result, they are unable to offer location-based services to our users as we desire. It is crucial for us to develop in-house custom LLMs that leverage our unique local knowledge graphs. We are building our own large language models (LLMs), that are based on top of Nextdoor’s raw content and the structured knowledge graph to power multiple products.

Please reach out to us if you are interested to learn more — we are hiring!

References

1. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
2. https://engblog.nextdoor.com/modernizing-our-search-stack-6a56ab87db4e
3. https://github.com/nmslib/hnswlib
4. https://engblog.nextdoor.com/using-predictive-technology-to-foster-constructive-conversations-4af437942bd4
5. https://maartengr.github.io/BERTopic/api/bertopic.html
6. https://openai.com/research/clip

From Pre-trained to Fine-tuned: Nextdoor’s Path to Effective Embedding Applications was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Nextdoor we build technology that empowers resilient, safe, and kind neighborhoods all over the world. Securing a product that empowers global communities requires diverse and inclusive teams, reflective of the communities we support.

Yet hiring and retaining the diverse talent needed to achieve our purpose remains an industry challenge. The gap is particularly evident in the cybersecurity field where 25% of the workforce and 16% of CISOs identify as female. According to the WiCyS State of Inclusion report 2023, women cite lack of respect and limited opportunities for growth in cybersecurity as top challenges accompanying lack of representation. We must keep working on it.

That is why Nextdoor welcomed the chance to celebrate diversity, alongside RSAC 2023, in Nextdoor HQ’s backyard this week and to partner with our neighborhood Women in Cybersecurity (WiCyS) Silicon Valley chapter. We are committed to building a diverse and inclusive workplace, and we are proud to work with organizations like WiCyS, who share the same values.

Nextdoor’s CISO TC Niedzialkowski kicked off with a warm welcome. CEO Sarah Friar framed the discussion by sharing how she launched her career by building a network at her first RSA conference as an equity analyst for Security Software at Goldman Sachs. She emphasized that diverse teams bring a variety of perspectives and experiences to the table, which ultimately leads to better problem-solving and innovation.

Moderator Ronit Polak, WiCyS Silicon Valley President, and CISOs Kathy Wang Lea Kissner Rupa Parameswaran Olivia Rose Jameeka Green Aaron, CISSP wowed the audience, covering everything from combating today’s top cyber threats including AI to imposter syndrome with incredible authenticity and humor. Closing us out Jameeka Green Aaron, CISSP called on WiCyS members to see themselves in the panelists and to thrive because representation matters!

Attendees ranged from aspiring cybersecurity professionals to a few celebrity leaders and practitioners from across, cybersecurity industry, government, and academia.

Learn more about Nextdoor’s initiatives to foster a holistically inclusive platform, and visit our careers page to see openings at Nextdoor.

Securing Diversity in Cybersecurity was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Nextdoor, our mobile applications on iOS and Android serve content to tens of millions of weekly active users. At this scale, we run a weekly release process for both iOS and Android, shipping hundreds of changes across multiple teams and dozens of mobile engineers.

Our team uses several observability processes and rollout strategies to keep these deployments safe and scalable. We most notably use phased rollouts to minimize the impact of a potentially bad release. Phased rollouts allow us to gradually increase the adoption of users for a new app version. For example, we can have a new app version be released to only 1% of users on the 1st day, 2% of users on the 2nd day, and so on. That way, if a new release were accidentally shipped with an uncaught regression, having it at 1% rollout means it affects fewer users, reduces its severity level, and gives us more time to react.

However, for many of our critical business metrics where a failure can sometimes be silent, most out-of-the-box observability approaches don’t work with phased rollouts. This is largely due to two problems:

1. Observability typically happens at an aggregate level. For example, we look at app sessions or revenue on a daily basis, across all users for a platform.
2. The behavior of early adopters on an app version differs from the median behavior of all users. Most importantly, early adopters are more active, almost by definition, to be in an early rollout of the new app version.

At Nextdoor, Daily Users are more likely to adopt releases over Weekly Users, Weekly Users over Monthly Users, and so on.

For example, consider an app session regression on a hypothetical iOS version v1.234.5 released March 4. If we had unknowingly introduced a regression where we didn’t count an app session 5% of the time, at a 1% rollout, our aggregate impact would be expected to be roughly 0.05 x 0.01 = 0.05% of all iOS app sessions, which is practically impossible to detect (read: noise) with aggregate-level observability. Even worse, early app adopters skew more active, which means that maybe we should expect 0.06% of all sessions impacted. Or maybe 0.07% of all sessions impacted. In short, it’s hard to tell exactly what our aggregate impact should be.

However, when iOS release v1.234.5 reaches full rollout in a week, a 5% app session regression would be business critical. We can detect the app sessions drop once it reaches full rollout by looking at week-over-week or month-over-month metrics, but by that point, several days would have passed.

Stacked graph. Top trendline shows our app sessions, which has a clear regression starting March 7 with a low point at March 14. Bottom trendline shows the release adoption over time due to phased rollouts.

How can we detect these issues on day 1, at 1% rollout?

A simple approach would be to normalize our business metrics to the total number of users on the release, and turn all metrics into relative metrics (e.g. on v1.234.5, app sessions per active user per app version). Unfortunately, as mentioned earlier, we can’t directly compare the app sessions from users who have adopted a release to those who haven’t as their underlying characteristics are too different.

What we’re trying to solve for these early adopters is: what is the difference between their actual app sessions after adoption compared with their hypothetical app sessions had they never adopted the release in the first place, or an unobserved counterfactual? In statistics, we can measure this through difference-in-differences analysis.

For iOS release v1.234.5, app sessions over time of users who adopted the new app release on March 4 (teal) vs app sessions over time of users who did not adopt (gray).

Difference-in-differences analysis is a simple causal inference method we can apply here to estimate this effect by accounting for the separate time varying effects of users that have and have not adopted a release:

1. For users who adopted the release, calculate the difference in their app sessions three days before and three days after the release period. In this case, we observed a -0.02 decline in app sessions.
2. Do the same for users that have not adopted the release. In this case, a +0.20 increase in app sessions.

Assuming trends would have otherwise remained constant (pre-trend assumption), we would have expected app sessions of release adopters to increase by +0.20 like we observed with non-adopters. However, they instead decreased by -0.02. We calculate the difference in differences to estimate a comparison against an unobserved counterfactual:

-0.02–0.20 = -0.22 decrease in app sessions due to iOS release v1.234.5

In practice, we don’t just calculate this in aggregate. We first make sure that our two cohorts exhibit similar behavior pre-adoption (pre-trend assumption). This is a critical step to difference-in-differences analysis. With a sample size over hundreds of thousands of users, we can achieve high confidence in similar pre-trend behavior with a simple standard deviation bound over the preceding few days to adoption. If this behavior holds, we then fit a linear regression model that estimates the average effect of a release for any particular metric:

y = β0 + β1* Time_Period + β2* Treated + β3*(Time_Period*Treated) + e

In the case of v1.234.5, we can measure statistically significant negative effects across multiple app sessions metrics.

Average % lift of metrics we ran App Release Anomaly Detection on for v1.234.5

With this difference-in-differences approach, we are now able to flag the app sessions decline due to v1.234.5 on March 5th, 10 days earlier than we normally would have been able to using week over week figures. We also mitigate the need to factor in external variables such as seasonality or day of week. This not only helps in diagnosing the source of the decline to a specific app release, it also isolates the regression to less than 1% of iOS users.

App Release Anomaly Detection allowed us to discover and fix the release regression at 1% rollout, before our aggregate observability even showed a drop.

App Release Anomaly Detection is one of the many tools we’ve built at Nextdoor to give us observability into our releases while iterating quickly. It is one of the foundational elements that allows us to deploy major app releases on a weekly cadence and have confidence in our stability. Operationalized, App Release Anomaly Detection has helped us prevent nearly all severe critical client-side regressions and gives us peace of mind to release bigger changes at a more rapid cadence.

If this type of cross-functional work between platform engineering and data science at scale interests you, we’re hiring! Check out our Careers page for open opportunities across all our teams and functions.

Written by Walt Leung and Shane Butler, with support from Hai Guan, Charissa Rentier, Qi He, and Jonathan Perlow.

Catching Anomalies Early in Mobile App Releases was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a thriving community, people are connected to their friends and local businesses. Nextdoor is the hyperlocal platform that mirrors these offline relationships. Every day, through active discussions on the platform, new relationships are formed and existing ones strengthened.

For example, a Nextdoor user can create a post like “I really like @XYZ cafe. @John is a hard working business owner and we should all support him by buying a cup of delicious latte!” Here, the post is created by at-mentioning (via the @ symbol) nearby businesses and users. From this post, users in the neighborhood can contribute by at-mentioning others to be part of the comment threads. As a result, John’s cafe thrives and acts as a neighborhood hub where new friends are made.

Every month, millions of these mentions are created in various discussions (including lost dogs!). In addition to posts and comments, a user can type into the search box and see, among other things, nearby users and businesses. All these features are powered by the same autocomplete service — a set of APIs to ingest data and handle typeahead search of different entity types (businesses, users, keywords etc) on Nextdoor.

This post focuses on how we built a proximity-based typeahead service to power typeahead use cases at Nextdoor.

Proximity-Based Typeahead Search as a Service

Any good search experience can be boiled down to two core components:

1. Relevance: Given a search query, whether the user sees relevant results or not. As a hyperlocal social network, relevancy is heavily weighted by geo proximity.

2. Low latency. Google Search found that

a 400 millisecond delay resulted in a -0.59% change in searches/user. What’s more, even after the delay was removed, these users still had -0.21% fewer searches, indicating that a slower user experience affects long term behavior.

For a good autocomplete experience, as users type, relevant results should show up instantaneously.

To meet the product requirements, we set out to build a service with the following design goals:

1. Low latency. There are hundreds of millions of entities on Nextdoor. The search latency at the service level should be less than 50ms.
2. Horizontally scalable to meet future scaling needs (we scale by adding more nodes).
3. Extensible. Typeahead search is a foundational API that enables other product features, so it should be easy to add other types of entities in the future.
4. High throughput for writes. We want to be able to index hundreds of millions of entities in a matter of hours.
5. Ease of operation and maintenance. When we index records, we should not impact production traffic.

Implementation

We landed on an in-memory-based solution that leverages geohash. At a high level, geohashing divides the earth into multiple zones based on latitude and longitude. It provides a good way to shard a large data set into buckets based on a zoomed-in level. Entities in the same bucket are in close proximity.

We used Uber’s open source geohashing library called H3.

For handling typeahead search, we decided to use sorted sets. This gives a set of benefits:

1. In-memory storage gives us the best possible latency characteristics for handling typeahead search.
2. It is easy to maintain. We can rely on redis persistence without having to handle durability ourselves.
3. By following the Command Query Responsibility Segregation (CQRS) pattern, we are able to index hundreds of millions of entities in a matter of hours with no impact to the serving of production traffic. Ingestion is handled by redis primary nodes in the cluster, and updates are then replicated to the read-only nodes which handle the typeahead search. Replication lag is less than 10ms.

With these two core pieces in place, we built a set of APIs that work together to handle all aspects of typeahead search:

* indexing_api (ingestion)

* typeahead_api (search)

* ranking_api (ranking by entity types)

Ingestion Path

Here is an example of the ingestion flow for businesses (Starbucks with id: 5, latitude: 47, and longitude -122):

For users, the typeahead search API works for both first- or last-name prefixes. Here is an example of the ingestion flow for users (Steve Jobs with id 4, latitude 47, and longitude -122):

Query Path

With the above structure in place, typeahead search retrieves results with a simple look-up using entity type, geohash key, and prefix. We then hydrate and rank the results before returning them to the client.

What we have today

The service has been running since August 2021. Every month we are handling hundreds of millions of typeahead search requests, and millions of comments with at-mentions are created. The service level for P95 search latency is less than 30ms.

Future work: typeahead for 1+ degree connections

To handle typeahead search with 1+ degrees (friends of friends), we can

1. Get a list of 1st degree connections.
2. For each user in the connection from step 1, get their connections.
3. Aggregate these connections and perform typeahead by prefix.

To reduce the network round trip between the first and successive calls, we can leverage Lua for edge computing.

Acknowledgement

It takes a team to move mountains! I would like to take the opportunity to give a shoutout to all the dedicated Nextdoor folks behind this endeavor:

Shivam Bhalla, Stephen Cheng, Yuki Mizuno, Rajesh Balasa, Siva Pandeti, Uzair Khan, Sharvil Parekh, Hung Dao, Josh Sibelman, Bojan Babic, Jane Wang, Sudhanshu Siddh, Omer Palaz, Kristy Duong, Tristan Eastburn, Paul Meng, Cory Dolphin, Andrew Munn, Tim Wong, Chintan Shah, Rahul Sureka, Madeline Neveaux, Murali Krishna Hosabettu Kamalesha, Glen Tona, and Avinash Chukka.

And by the way, we are hiring!

Typeahead Search at Nextdoor was originally published in Nextdoor Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.