Build an offline-first to-do app with React + CouchDB
In this tutorial, we are going to build a to-do app that can continue to work when the user loses their internet connection. The use cases for this type of functionality are countless, so it’s a good trick to have in your arsenal. The number one thing to know when approaching this type of problem is how multi-leader replica sets work. Let’s dig into that first.
Multi-leader replica sets
To understand this concept, let’s go back to the basics. How do we persist data? We store it in a database.
How do we write to the database? We make a function call to a node that has the authority to write to said database. We call these types of nodes leaders.
How do we read from the database? We make a function call to a node that has the authority to read from the database. All leader nodes can read from the database. Nodes that are not leaders are called followers.
How do we create multiple nodes that have read/write access? We create replicas of the database and assign each node to one copy of the database.
What do you call a system that has more than one leader? A multi-leader replica set.
So what do multi-leader replicas have to do with my to-do app?
Let’s imagine for a moment a more “traditional” architecture where your to-do app pings some API endpoint to perform a CRUD action (create, read, update, delete). What’s going to happen when a lack of internet connection stops the network call from succeeding? If we go down the rabbit hole, we may come up with ideas such as creating an offline queue system that continues trying to batch upload the failed data until it receives a success response. Even if we succeed with this endeavor at first, we’ll now be forced to maintain this tool, making sure all our API calls have failover logic that adds the failed data to the queue. In addition, as the database complexity grows, the queue may need to apply batch operations to multiple tables once the device is back online. As you can see, we’re now essentially in the business of creating a web database storage engine.
If we double-down on the fact that we need to maintain a copy of the data as the user continues to apply CRUD operations despite being disconnected from the remote database, why not just keep a copy of the database in the browser and write to it instead of the remote one?
- The semantics from a developer perspective are simpler: a developer just needs to focus on ensuring the data is getting written to the local database.
- When the internet goes offline, there’s no issue. We were writing to the local database anyways.
- Since we are now dealing with a multi-leader replica set, approaches to creating a stable solution are known and documented. Additionally, there are existing tools that can help us accomplish this multi-leader scenario.
Roll up your sleeves
Now that we’ve covered most of the theory, let’s implement our app. Our implementation will consist of three steps:
- We’ll setup up a UI-only CRUD app in React
- We’ll create a database that lives on the user’s device
- We’ll create a remote-hosted database that syncs with the database that lives on the user’s device
UI-only CRUD app
- Clone this repo: https://github.com/alien35/offline-todo-app
- Check out the branch step-1/basic-ui, also available here: https://github.com/alien35/offline-todo-app/tree/step-1/basic-ui
I’ll spare you the details. There’s nothing too note-worthy in this step so let’s move on.
Create a database that lives on the user’s device
Check out the branch step-2/add-local-db, also available here: https://github.com/alien35/offline-todo-app/tree/step-2/add-local-db
If you take a gander at package.json, you’ll see that we’ve imported the npm package pouchdb. PouchDB is a powerful NoSQL web datastore that will enable us to achieve our multi-leader goals. Learn more about its awesomeness here: https://pouchdb.com/
The meat of our database logic lies in the custom hook defined here: src/utils/useDatabaseMetadata.tsx.
Take a moment to get acquainted with the code and then let’s discuss a little more theory.
Ordering Events
The same way eventual consistency is a hot topic when dealing with reading data from a replicated database, causal consistency is a big topic for writing data to a replicated database. In our to-do app, it’s trivial enough to build our solution around the idea that the user will only be using one device at a time, but what happens if they’re using two (or more) devices at the same time? Focusing specifically on ID generation for out to-do records, using a time-base ID generating process could foreseeably lead to the two devices accidentally trying to create two new records with the same ID. This is a possibility particularly in our web-based solution since the Javascript Date object only has millisecond resolution.
Logical Clocks
The standard approach to managing ID generation in a concurrent system is to rely on logical clocks rather than a time-based clock. Logical clocks measure the relative ordering of events in order to help us achieve causal consistency. In particular, the Lamport Timestamp is a simple but solid approach to implementing logical clocks a in distributed systems.
Lamport Timestamps: Defined
In a single-device scenario, you could simply keep a counter on the device that increases with each write event. If you were to run this app on multiple devices though, you would quickly start running into conflicts as the second device isn’t aware of the first device’s counter value and so you’d end up with overlapping IDs. A Lamport Timestamp accounts for this by assigning each node (remember that in our case, each device is a node) an ID and then the database record IDs are generated by combining the node ID with the event counter.
So how do we build this?
Let’s look at src/utils/useDatabaseMetadata.tsx again
In fetchMetadata() we first check to see if the device has already been registered as a node. If it hasn’t, we generate an ID for the device and then store the metadata around the node ID as well as the “event count” that we’ll use for our logical clock at _local/database_info. The “_local” prefix is just a technicality to prevent this device-specific metadata from being replicated once we set up a cloud version of the database.
If you look at src/Todos.tsx, you’ll see we’ve elaborated our CRUD app to update our PouchDB database with the corresponding write/read/update/delete actions. You can see our Lamport Timestamps being generated on the fly here https://github.com/alien35/offline-todo-app/blob/step-2/add-local-db/src/Todos.tsx#L46
Syncing our local database with a remote-hosted one
The first thing we need to do is setup a remote-hosted database. We’ll be using CouchDB as there’s built-in functionality to sync with our PouchDB copy of the database. Instructions can be found here https://pouchdb.com/guides/setup-couchdb.html
Tip: Make sure to following the CORS instructions on that page, and also log in at http://localhost:5984/ to ensure our browser app has access to the database.
The final stretch
Once you have that setup, we’ll officially be at the final stretch of building the app. Check out the branch main.
The note-worthy bit of this branch is that we’re now syncing our local database with the remote one. The logic for that can be found here:
https://github.com/alien35/offline-todo-app/blob/main/src/utils/useDatabaseMetadata.tsx#L17
PouchDB lets us simply say that our source database, the local one, needs to be replicated to the target database, the remote one.
Disable the internet connection
To see what we’ve done in action, in your browser’s Developer Tools go to Network and select Offline.
Add a to-do or two and monitor the Network. The Network should now start getting clogged with failed requests:
That’s because even though we’ve written to our local database, our local database is trying to sync with the remote one and is unable to connect. Re-enable the internet connection, and now the requests should start succeeding again:
Unrelated PSA: Looking for a new high paying software development job? Send me your resume to alexleondeveloper@gmail.com and I’ll get back to you!
Conclusion
In this tutorial we reviewed concepts from multi-leader replication and applied them in order to build a to-do app that works even when the user is offline and syncs up again with the cloud when the internet connection is restored. We overviewed logical clocks — Lamport Timestamps in particular — as a simple but solid approach to managing IDs in a distributed system. As per usual, all the source code is available here:
