Local-First

Local first means that the app works also offline, and even more, that it primarily works offline, and that the syncing that happens once we are back online, is transparent.

This is a paradigm shift from the current Cloud architecture, as most apps that are developed nowadays cannot run without internet connectivity. And the situation today is a complete reversal from the one we had at the end of the previous century, where all apps used to work only offline.

After the conversion of offline-only apps (the “.exe” apps), into SaaS (Software as a Service) apps that run in the cloud and cannot work without internet, we are now going towards apps that are capable of both: online and offline. This is what we call Local-First apps.

It is a very hot topic and the first big conference on the subject happened in Berlin at the end of May 2024.

In order to work properly, a Local First app needs to use CRDTs (Conflict-free Replicated Data types) that are a special way to encode/save the data, that stores additional metadata with each record, in order to help solving eventual conflicts at the moment of the merge/sync.

Some rules are agreed upon in advance on how to deal with the conflict, based on the metadata. And those rules lead to a deterministic and consistent conflict resolution mechanism across replicas, regardless of the order in which the updates are applied.

The best CRDTs out there are based on a DAG of dependencies (Directed Acyclic Graph), that encodes the causal past of an update/operation/commit (all synonyms).

Each operation indicates which previous operations it “sees” in its “causal past” at the moment when the operation is committed.

Other concurrent modification can happen, that might not have the same set of causal past, or that can have the exact same one. In this case, we say that a fork happened.

When the syncing/merging occurs, knowing the causal past for each commit that needs to be merged, enables the local replica to be sure that it didn’t miss some previous commits. Also, this causal past is important to verify the correctness of the commit. A commit can have rules that need to be enforced. But those rules can only be checked based on the causal past that was present at the replica at the moment when the commit was created, and not including any other commit that might have happened concurrently.

On this topic, I let the reader refer to the work of Martin Kleppmann on merging protocols and preserving invariants, and to the “keep CALM and CRDT on” paper.

To come back to the higher level overview, local-first apps deal well with offline and online collaboration with real-time editing.

Be it a document that is shared between the different devices of a single user, or a document that is shared among several users (and their respective many devices), the user will always be able to view and/or edit the document while being offline, and then sync with the other replicas after regaining connectivity.

This by itself, is a big paradigm shift for the developers, who are more used to calling remote APIs to access data.

Here the data is always accessed locally (in this sense, the backend sits in the “client” or in the “front-end” if you prefer).

It is the NextGraph protocol that deals transparently with the syncing mechanism. The developer does not need to deal with it. We provide access to a local API (in javascript by example) and data reads and writes happen on this local API.

We will come back to our syncing protocol further down, but before that, we would like to explain our P2P network and why everything is encrypted.