Design

In the previous chapters, we already introduced the concepts of :

We will now dive a bit deeper on some other core concepts and the rationals behind them, and then on the internals of the protocols and formats.

History of the project

until August 2024

I am carrying this project since 15 years. I carried it sometimes in my heart as a wish, sometimes in my mind, with some ideas and thoughts, and sometimes on my shoulders, with experiments and coding, specially in the last 3 years.

The first idea goes back 15 years ago, and it was about data-first and malleable/composable software, very much inspired by the vision of Tim Berners-Lee about Linked Data and the Giant Global Graph (GGG) that he envisioned back then, as the successor of the WorldWideWeb. At the time I was working in big-data startups.

In 2013, the Snowden revelations confirmed the suspicions we all had, about mass surveillance and the absence of privacy online. It is at that moment that I looked closer into security and encryption. I also quit my job at that time and started to dedicate more time toying with some prototypes.

In the following years, I made a first attempt at building a system based on graph databases (JanusGraph) and I bumped into several issues, specially about consistency of the database in a decentralized system. It turned out that Java wasn’t the best language for what I was doing, and that Property Graphs weren’t neither. I had originally discarded SPARQL and RDF because it looked rebutting, but after playing with WikiData model, I understood it was very powerful and I then fully embraced RDF and SPARQL in 2019.

In Early 2020 I attended the first P2P festival in Paris where I expressed my desire to explore the convergence between Semantic Web and P2P technologies. At the same time, I met with the people of Virtual Assembly in France which is an association that acts as a hub of many projects related to Semantic Web and Ecological Transition. As I am fond of both subjects, it was a match instantly.

They presented me with the new project they were working on: the refactoring of SemApps, a toolkit for Semantic Applications. I joined them and implemented the ACL module (WAC-WebACL) inside a plugin of Jena, the triplestore that SemApps is based on for now.

In 2021, I did a quick test of IPFS and IPLD, and realized it wasn’t suitable for what I envisioned, and then I got acquainted with CRDTs and understood it was the key I was missing so far. Still, I couldn’t see any framework or software that was already offering such a combination of Semantic Web, P2P, CRDTs, and E2EE. At the time, SemApps was getting more and more involved with ActivityPub, under the direction of Sebastian, and I contributed a bit to that too, before focussing my energies entirely on NextGraph, that I founded at the end of 2021.

P2Pcollab and Lofi.re

At the same time, at the end of 2021, I was redirected to a small group of developers who had gathered on similar technologies, but had recently disbanded. Some of them had received grants from NLnet, a funding organization that I didn’t know until then. I proposed the project NextGraph to the former members of that group that had disbanded, but the chemistry didn’t work. Eventually I stayed in touch with T.G. of P2Pcollab, who was one of them, and who had researched P2P networks and decentralized data sharing before.

In the first half of 2022, he was working on a draft design that he later named Lofi.re and I partly contributed to this design. He also transferred to me all his knowledge about P2P network and I learned about his design too. T.G. had a grant to finish and I helped him with the code in October 2022, as I was learning Rust, that I insisted should be the new language of this implementation, because I had already decided it would be the language for NextGraph. Eventually, the project Lofi.re stopped and I applied for my own grant at NLnet, which led to the present work.

As a consequence, the design of NextGraph network protocol is partly inspired by the previous work of Lofi.re by T.G. of P2PCollab, of which I took several components. I was familiar with his work, thanks to our previous cooperation, during which I contributed to the design of the immutable blocks and convergent encryption.

Several concepts that are implemented in NextGraph, come from Lofi.re: the 2-tier topology, the overlays, the data chunked in blocks, the pub/sub mechanism, the organization of repos into branches.

But unfortunately, this previous work in its whole, was flawed in several ways, and I had to revise it, correct it, improve it and adapt it to the needs of NextGraph, and confront it with real use-cases. The design was too abstract, and unsound in some parts. Eventually, NextGraph protocol is not compatible with the draft protocol of LoFi.Re.

What we didn’t take from this draft design: the explicit ACKs, the permission system, the signature system, the core protocol (as it claimed to implement LoCaPs but wasn’t effectively doing it), the sync protocol, the binary file format, and more.

More features were added, like the distinction between Inner and Outer overlays, one overlay by Store instead of one Overlay by Repo, the ReadCap for an overlay and repo that is very different from the secret_keys of the original design, the refresh mechanism for capabilities and epoch (which wasn’t specified at all in Lofi.re), the threshold signature mechanism with certificates, etc…

Overall, I want to thank T.G. for his good contribution with his draft design (which was supported by another grant from NLnet), and for sharing with me his good intuition and knowledge which made the network stack of NextGraph possible.

With NextGraph, we hope that P2Pcollab and all the people interested in the previous work of T.G., will find a suitable implementation of those ideas, that would match their own needs. Feel free to contact us and let us know if NextGraph suits you.

Now after this introduction, let’s dive into the architecture of NextGraph