The history of databases at Netflix: From Cassandra to CockroachDB (2024)

The history of databases at Netflix: From Cassandra to co*ckroachDB (1) [ Webinar ]

A Netflix engineer explains...

How Netflix unlocks analytical use cases using change data capture and co*ckroachDB

Watch now

In 2008, after Netflix pivoted from DVD-by-mail to streaming, they were running the streaming service on premise and suffered a 3-day outage. That was the beginning of their move to the cloud. First they moved to AWS. Then, in 2014, they adopted (and popularized!) Cassandra to support their need for global replication.

In this two-part presentation, Netflix Senior Software Engineers Shengwei Wang and Shahar Zimmerman explain why Netflix has adopted co*ckroachDB and how they’re deploying it.

Three Netflix use cases that don’t fit Cassandra

In 2019 Netflix began to run into problems with Cassandra for certain use cases. At that time Netflix was seeing an increase in demand and an increase in their database requirements. For example, the studio side of Netflix was producing more shows and movies which added new data needs. Generally speaking, there was an increase in the need for consistent data, specifically for these three use cases:

Cloud drive service: a file system-like service for media assets which was needed by the Netflix studio side of the business.

Content delivery: Netflix built its own CDN called Open Connect, and they needed a control plane service to manage network devices around the world.

Spinnaker: a continuous delivery platform on the cloud.

These are all global services that need to support consistent transactions at times. Supporting consistent transactions is problematic with Cassandra, because in Cassandra you don’t get rich transactions. Instead, you have lightweight transactions, which are extremely performant, but limited. Also, the secondary indices in Cassandra are unreliable and often don’t work.

The other options Netflix considered for these use cases were AWS Aurora as well as AWS DynamoDB but they ran into some limitations with scalability there.

Netflix decided that they needed a scalable SQL database for those three use cases and established a set of requirements: Multi-active topology, global consistent secondary indices, global transactions, open source, and SQL.

The history of databases at Netflix: From Cassandra to co*ckroachDB (2)

As you can see from the graphic, co*ckroachDB satisfied each of these requirements and earned a seat at the Netflix architecture table.

How is co*ckroachDB deployed at Netflix?

The history of databases at Netflix: From Cassandra to co*ckroachDB (3)

In 2020 Netflix deployed their first co*ckroachDB cluster in production. Today they have 100 production clusters and 150+ test clusters. At this time most of the clusters are deployed in a single region with three availability zones. The biggest co*ckroachDB cluster at Netflix is a 60-node, single-region cluster with 26.5 terabytes of data.

In this section of the presentation, Shengwei Wang explains why Netflix does not use the co*ckroachDB binary, how Netflix navigates upgrades to new versions of co*ckroachDB, and how Netflix deploys/manages co*ckroachDB. You can get all those nuanced details directly from Shengwei right here:

More co*ckroachDB at Netflix resources

Since their adoption of co*ckroachDB in 2020 the Netflix tech blog has published a couple different articles about different use cases that leverage co*ckroachDB:

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

Data Mesh — A Data Movement and Processing Platform @ Netflix

Unlocking analytical use cases at Netflix with co*ckroachDB

If you’re interested in learning how DoorDash, Booking.com, Santander, and other companies use co*ckroachDB you can find their presentations on our RoachFest home page, and you can read more case studies on our customers page.

The history of databases at Netflix: From Cassandra to CockroachDB (2024)
Top Articles
Latest Posts
Article information

Author: Frankie Dare

Last Updated:

Views: 5801

Rating: 4.2 / 5 (53 voted)

Reviews: 84% of readers found this page helpful

Author information

Name: Frankie Dare

Birthday: 2000-01-27

Address: Suite 313 45115 Caridad Freeway, Port Barabaraville, MS 66713

Phone: +3769542039359

Job: Sales Manager

Hobby: Baton twirling, Stand-up comedy, Leather crafting, Rugby, tabletop games, Jigsaw puzzles, Air sports

Introduction: My name is Frankie Dare, I am a funny, beautiful, proud, fair, pleasant, cheerful, enthusiastic person who loves writing and wants to share my knowledge and understanding with you.