Transactions
We want to address the question of the database and consistency model.
We have seen earlier that updates on the data happen inside Commits. Each commit represents a transaction, that is executed atomically when received by a replica. Those transaction are called asynchronous.
Synchronous transaction
As we explained above, NextGraph is based on CRDTs and offers strong eventual consistency guarantees.
That’s a paradigm shift, and developers will have to learn how to code new applications that access the data locally instead of remotely.
There is plenty of work needed in order to adapt the many open source applications already existing, so they can fit the CRDT mechanism. For apps that are oriented towards content editing, CRDT would be a perfect match and it will bring the feature of live/real-time collaboration and automatic syncing to any such app and content editor (as long as the data format can be based on JSON or XML or RDF).
That would work well for everything that is content/document oriented.
But in some cases, the App is dealing with some business logic that needs ACID properties on the underlying database.
By example, any e-commerce app that is selling something online, needs at some point (when checkout is happening and stocks need to be checked and updated) to be able to run an atomic
and immediately consistent
transaction on the database. CRDTs are everything but that.
It wouldn’t make sense that developers using our Framework would need to install and interface with a Postgresql server for any transaction that should be ACID.
For this reason, we have integrated a specific feature into the Repo mechanism, that lets the developer create a Synchronous Transaction in a document (a document that should be understood as a database). Those types of transaction have higher guarantees than the Asynchronous transactions based on CRDTs, as they guarantee the finality of the transaction, which means that once this commit has been accepted by a certain threshold of Signers, it is guaranteed that no other concurrent transaction has happened or will ever happen that would conflict with or invalidate the invariants. Therefor the commit is final.
Basically what we do is that we temporarily prevent any fork in the DAG while the transaction is being signed (we also call that “total order”. it creates a temporary bottle neck in the DAG) and the threshold that must be above 50%, guarantees that any concurrent or future fork will be rejected. Supermajority can be used to calculate the threshold, in cases where Byzantine or faulty users have to be considered. (given N = all users, F = byzantine and/or faulty users, supermajority = (N+F)/2 +1).
This is synchronous because it requires that the quorum of signers is online at the same time, and can agree on the transaction.
For simple use cases, a single Signer can be enough, as long as we know it will be always online (and it can run by example on a self-hosted broker or in our SaaS). For the stock management of a small online retail business, it will most probably be enough. This unique signer will act as an ordering service that will force a total order on the transactions.
If more reliable setups need to be created, a pool of signers can be added, with a quorum that is only 70% of the pool, by example. This setup allows some of the signers (30%) to fail or be offline, while still being able to provide the total order guarantees.
Furthermore, if the integrity of the database (understand, the Document) needs to be checked and asserted by several parties with conflicting interests, then those parties have to setup one Signer each, and the quorum should be 100%.
It is also possible to configure the Repo to have parts of its data subjected to the total order requirement (ACID), while other parts of the data being subjected to partial order (CRDTs).
In addition to traditional DBMS ACID business logic required by most e-commerce applications, this feature also enables the creation of Smart Contract that can represent a Finite State Machine (FSM) that is distributed among participants, and signed by a quorum of those participants. More on that in the Smart-Contract chapter.
There is a limitation : synchronous transactions cannot cross the Document boundaries.
For this purpose, we have designed a mechanism of cross-document handshake, where 2 Documents exchange messages (commits) that are signed by their respective quorum, and this mechanism enables cross-document synchronous transactions. You’ll find more on this topic in the Smart-Contract chapter.
This feature is very important, as it will enable Documents to hold data that has ACID requirements, mostly used in business logic. It complements well the content oriented feature of CRDTs, while providing a unified Framework for developers who can opt for total-order or partial-order according to their needs in term of consistency and finality.
Synchronous transaction are already implemented and we use them for managing permissions, and compact operation. But they are not yet available in the API. Stay tuned!