Documents, Stores and Blocks

Everything in NextGraph, every data, is stored inside a Document.

That’s a central concept, that is not that hard to grasp, and that has been adopted in database like MongoDB and CouchDB.

A Document contains some content (that varies with the format. In our case, RDF, JSON, plain text or XML). All the Documents of a user form a local database that can be queried and updated within transactions.

Each Document transparently synchronizes itself between all your devices, and with the users that you shared it with (if any).

Dual nature of a Document: Sun vs. Cloud

Each Document has a double nature :

It has a Document-like nature where you can store and edit some rich-text, plaintext, or just some JSON or XML data, according to your needs and the primary class of the Document that you have chosen when you created it. This is so far very consistent with what we expect from a document in general. All the apps that you use or develop with NextGraph, will store their data inside these Documents. The Document-like nature is represented with a “cloud” icon. A note for developers: In the internals of NextGraph, the Document-like nature of a Document is called “discrete”, because otherwise it is too many things that are called “document”.

A Document also has a Graph nature, which is something new that NextGraph added (hence the name “NextGraph”). This “graph nature” of the document lets you link this document to other documents. It also lets you enter some specific data or meta-data about this document, that will be part of the Graph of all your documents. This graph is something important, that you are not used to. Social networks are all based on Graphs. The Web itself is a huge Graph. When you follow or get followed, when you like or comment on a Post, when you write a DM to someone, all this information is stored as a Graph, that connects different documents together. Later, you will probably need to query this graph. This is done transparently when you want to see all your followers, and when you want to consult the stream of all the posts that they have published recently, or when you want to search for something. It is also used for recommendations. But in any case, what is important to understand is that internally, each Document can be linked to any other Document (like the classical Web that links webpages) and that the applications you will use on NextGraph, will also store more Graph information. Because NextGraph is local-first and decentralized, this graph information is available to you at all time. And if you want to see it, you can go to any Document and in the Document Menu, you can select “Graph” and you will see options to view and edit the Graph. This is a bit technical for now (you will see things like Turtle, SPARQL etc) but in the future, we will provide here some nice tools where you will be able to explore your own graph easily. The Graph-like nature is represented with a “sun” icon. The Graph part of the document is stored in the RDF format.

In addition, you can also attach some binary files to any Document. Those files are immutable in the sense that you can add them or delete them, but you cannot modify their content.

So to recap, in a Document we have :

  • the Graph part (some RDF triples). This is mandatory and always available in all documents, even if left empty.

  • the Discrete part (some JSON, XML or plaintext. Based on Yjs or Automerge CRDTs). This can be optional, meaning that it is possible to create a purely graph-based document that doesn’t hold any discrete part.

  • some optional binary files attached to the document.

If you are curious about why we implemented this dual nature of documents, read more about that in the CRDTs chapter.

Repository

Each NextGraph document is stored in a separate repository, that holds all the changes (that we call “commits”) and also all the permissions. The commits of such repository mutate both the RDF resource and the discrete document, and can add and remove binary files.

Each Document is identified with a unique identifier, that is a public key randomly generated at creation time. This ID will never change over time. Internally we call this a RepoID.

It is a permanent ID of the form did:ng:o:[44 chars of the ID] by example did:ng:o:EghEnCqhpzp4Z7KXdbTx0LkQ1dUaaqwC0DGVS-0BAKAA.

This Identifier is a valid URI that can be used to link a document with another (both in the rich-text or in the RDF data, as subject, object or even predicate if a document is used to define an ontology). We call it Nuri, for NextGraph URI.

It follows the DID (Decentralized Identifier) format, and an adhoc DID Method will be specified and registered shortly.

Type of a Document (primary class)

When you create a new Document, you first have to select its type, which is also called “primary class” internally.

There are already several types proposed to you (not all of them are currently implemented. See a list in features ), and App developers can add new types.

Here is a short list of the most used types :

  • Post or Article : a rich text document that can contain links, images, bold, italic, embeds etc…
  • Markdown : the same, but using the MarkDown format.
  • Plain Text : just some text.
  • Contact : represents a person you know or about whom you have contact details.
  • Event : An event that has a date and a place and that other people are invited to.
  • Form : a form that needs to be filled-in.
  • Spreadsheet : columns, rows and cells to store numeric and text values, with formulas.
  • Chart : a visual representation of some data
  • Geo Map : a geographical map that contains some markers
  • etc…

In order to create a new Document, you can use the “plus” button on the main screen of the App.

Viewer, Editor and Apps

A document can be opened in

  • Viewer mode or in
  • Editor mode (if you have permission to edit the document, of course)

A document always opens in Viewer mode (read-only) and if you want to modify it, you have to go to the “Document menu” to select an Editor. The “Document menu” is accessible from any Document, by clicking on the “3 vertical dots” button at the top-right corner of the screen.

For now, only one editor is provided for each Document type, but in the future, you will be able to install more Editors, and also more Viewers.

If you open the same document in 2 different tabs or browser windows, or with different apps (on mobile, desktop, etc), and when you edit the Document in one of the windows, it automatically updates itself in the viewer or editor that is opened in the other window(s). Try it! it works great.

But please pay attention to the fact that by default, the live editing mode is not activated, and instead, you have to click on the “blue mark button” that appears at the top-right corner of the screen, in order to save your changes.

If instead you want to switch to live editing you can do so with the toggle button of the same name, accessible from the Document Menu.

If you are editing a text document, please pay attention to the fact that live editing is going to send each character one by one, and this has a huge data size overhead/amplification, as each character will be encrypted and sent in our network, separately.

Viewers and Editors are what we call “Apps”. And they can be installed from the built-in App Store (not ready yet, but will be available soon).

NextGraph comes with a set of Official Apps (listed in Features) that are always available (they don’t need to be installed), and developers can also create new apps, that will need to be installed by the final user (but when the user receives a link to a Document for which they don’t have the necessary app installed, we will propose them to install this missing app automatically).

History

Each document keeps track of the whole history of modifications. This is accessible from the History pane, from within the Document menu.

You will be able (in the future) to select a specific revision and “revert” to it. Or to see more details about each modification (that are called “Commits”).

Because NextGraph is local-first and based on CRDTs, if you modify the Document concurrently from 2 different devices while being offline on one or both of the devices, then a temporary “fork” will happen on the Document History. This is visible with a new colored branch that will pop-out of the blue branch.

Once the two devices will be back online, you will see that they will automatically synchronize themselves and the “fork” will be automatically “merged”. The colored branch will come back to the blue main branch. This is what CRDTs are all about. It syncs and merges all your changes, without any conflict, so it can do that transparently without asking you what to do (unlike “git”, for those who know what “git” is).

You can try this between 2 tabs in your browser, or a browser tab and a native app, or 2 different native apps (desktop and android by example). It works great!

Blocks and Branches

We just explained that a Document has a type (a primary class) and that it keeps all the changes (called Commits) in a history.

To be more precise, this primary class and all those commits, are not stored for each Document, but for each Branch within a Document. What is a branch ? And why can a document contain several branches?

A branch is a concept that helps us separate the data within a single document, and it can be used in several ways.

  • A document can be composed of several blocks. Imagine the blocks like separate paragraphs in a text document, or separate embeds (video, table, spreadsheet, etc…) of content that come from other documents or that have different types, and you want to assemble all those blocks into one document. So you select those blocks and put them one after another. When the reader opens a document, they will not see that it is made of several blocks. To them it will just appear to be one document that they scroll into.

  • We also use branches when we want to fork the work that is happening on a Document (let’s say in collaborative editing). This can be handy if we want to keep the current version of the Document unchanged, while we are working on a newer version. All the work on the newer version is not visible to the reader, until the “working branch” is merged back into the main branch. This is something that programmers will understand very well, but that can be very handy for everybody else too. Please note that we use the terms “fork” and “merge” here as we did when we talked about CRDTs earlier, but it is not exactly the same thing. Here the fork and merge happens only on request from the user. You can decide at some point to fork a document or block. Then you can also decide that it is time to merge it back. But that’s a bit different than the automatic and transparent “merge and fork” that is continuously happening every time that someone goes offline. To make things more clear, we don’t call the later this way, we just say the document synchronizes itself. Fork and merge should be terms used only when the user decides to manually fork.

  • Branches and blocks can also be used to store different translation of your content. You do not want to create a new document for each translation. Instead, you create separate blocks within the same document, and each block holds the translated content. NextGraph is designed to help you with managing translation and multi-lingual content, and when the user will open the Document, the most adapted language will be selected for them.

  • Another advantage of blocks, is that it lets you create different types of content within the same document. As we have seen earlier, NextGraph offers different types of content (called primary classes). We said earlier that you have to select a type when you create a document. That’s true. And this will in fact be the type of the main branch of your document. But then you can add more blocks, that will have different types. So you can combine many different types of content inside the same document. And eventually, you can even change the type of the document itself, by switching the main branch to be another branch (more on that later).

  • Also, we internally use the branches feature to store some internal data that you don’t see, but that needs to have different branches for the good functioning of NextGraph.

One of the advantage of dividing your content into several blocks is also that blocks can have different read permissions. So if you want to share some information with a certain group of people, but not with some other group, but still refer to that as one document, then you can use blocks. By example, let’s say you have a Social profile, where you describe yourself, your interest, and where you put some contact information. Maybe the contact details should not be available to everyone (like phone number, postal address, etc..) Still, you do not want to create 2 documents for that purpose. So you will have 2 blocks. One will only contain the minimum public information you want to share to everyone. The other block will contain more details like phone number and address, and it will include also a link to the other block. This way, you don’t have to copy paste the general information about you in both blocks. The “include link” is transparent for the privileged reader who got access to the more private profile. They will not see that the document is composed of 2 blocks. To them it will just appear as one document. And of course, all the other readers who only got the link to the generic block, will not see that there exists another block that contains more private details.

So, to recap. Each branch holds a separate list of commits, that can be seen in the History pane.

For now the App does not let you create new blocks and branches, but it will come very soon. Internally, the branches are already there, and they work well. What you see for now in the App when you open a document, is called the “main” branch. Later, you will be able to add more blocks and branches.

Those who are more curious about the internals of the branch, commits, DAG (Directed Acyclic Graph) used in NextGraph, can refer to the Repo format of the specs.

Stores

Now let’s dive into the last important concept related to Documents in NextGraph: the Store.

A store is a way to group several Documents together.

When you created your Wallet, we have created for you, inside your wallet, a personal identity. You can also later create more identities if you wish to (not implemented yet).

Inside your identity, we have also created 3 stores by default for you!

Those are :

  • the public store : is equivalent to your website, blog, or public profile on social networks. You can put in this store all the documents, pictures, videos, posts, data, etc… that you want everybody to have access to, without the need for special permissions.

  • the protected store : is a space where you can share data, documents, and media with other users, but they will need a special link and permission in order to access them and collaborate with you on those documents (or just read them). You can also create Groups here when you want to chat or collaborate with several users. This store also acts as your protected profile for social networks. Only your followers will be able to see this profile, as it is not public.

  • the private store : this is a place where you put only private and personal information that only you have access to. This store will be accessible to all your devices, so it is useful in order to synchronize private data between them. You can put sensitive data too, as everything is encrypted. Nobody else will ever see this data. It is not possible to share the documents of your private store with anybody else.

You can also later, move documents from one store to another, if you are the owner of such document.

When you arrive in the App, the first screen that you see, and this is your Homepage, is in fact, your private store, where you can organize your personal data.

Then you can access your protected store with the icon that represents 2 users.

And your public profile/store can be found with the icon that represents one user.

We said that you can create Groups when you want to share, exchange and collaborate with other users. In fact, each Group is a separate Store. This is helpful because each group can have its own set of permissions, and then you can configure the store so that all the documents included in this store, inherit the permissions of the store. This way, we can manage the group easily. If we add a member to the group, they immediately get access to all the documents in the group. The same happens when you remove a user from the group: they loose access to all the documents at once.

Because most of the time, collaboration on Documents happens within a group of users that will probably share more than one document between each other, the Store regroups all the Documents that such Group wants to interact with. Not to confuse this Group with the E2EE group i was referring to in the Encryption chapter. In fact, in NextGraph terminology, we never talk about any E2EE group. Instead we call that a Repo, or a Document. A Repo is basically the equivalent of an E2EE group for one and only one Document. So when we talk about a Group, we are referring instead to a Store that gathers all the users and their peers, and where they will be able to share several Documents (or Repos if you prefer).

As you will see, a Store can be organized into folders and sub-folders, making it very easy to keep tidy. It is the equivalent of the “drive” that you have been using in cloud-based sharing system.

Write permissions are managed at the level of the Document, not at the level of the branch or block. But read permissions can be by block or branch. See more about the permissions here.

We also included the feature of having a chat system inside each document. The chat is only accessible to the editors of the document. It is helpful for coordinating the editing work while working on a collaborative document. (not implemented yet).

Another feature that is planned for the documents, is to have the ability to comment and annotate on every document.

Finally, there is an additional type of Store, that we call “Dialog” and it is there to hold all the data you exchange with another user (and only with that other user). We use it to store the DMs (direct messages) that you exchange with that other User, and it can also be used to share and create documents with that other user. It is like a private drive, just for you and the other person. You cannot add more users to this store. It is a nice way to organize your documents and content shared with that other person, because scrolling the whole chat is not fun when you search for something that you have exchanged 1 year ago. Here with the Dialog store, you can keep and organize all this content in a more permanent way, and easy to find it later on.

With all this description of documents and stores, you can already understand that NextGraph is also a Social network ! Let’s dive into that more in details .