App Protocol
All our protocols and formats use the binary codec called BARE.
The App Protocol lets the Application talk with the Verifier.
This protocol exchanges content that isn’t encrypted.
In the native apps, this protocol runs between the front-end part of the App and the back-end part that is running inside Tauri from Rust compiled into the app binary. Both sides of the protocol are in the same process, but there is an interface between them provided by Tauri (which is made of JSON objects), but this is transparent to the App developer.
In the web-app, the same happens, but the “back-end” part runs inside the same JS context, and is some WASM code compiled from Rust. The passing of messages doesn’t involve JSON, as JS POJOs are directly exchanged at the interface.
If the App opens a session with a remote Verifier, the protocol is ran inside a WebSocket, with Noise encryption.
This protocol is available directly from the JS and Rust APIs and you do not need to implement it again or send messages manually. New bindings are always welcomed, for any contributor who wants to add them.
All the protocol messages are embedded inside an AppRequest
message, and are replied by the Verifier with an AppResponse
.
Some Requests will respond with a stream of AppResponses, while others will respond with only one.
Streamed responses :
- Fetch(Subscribe)
- FileGet
A new session needs to be opened with the LocalBroker methods, not documented here (it will be documented in the APIs reference). Once the session is opened, the session_id needs to be passed in every request.
For a reference of the common types, please refer to the Repo format documentation
AppRequest
Request
enum AppRequest {
V0(AppRequestV0),
}
struct AppRequestV0 {
command: AppRequestCommandV0,
nuri: NuriV0,
payload: Option<AppRequestPayload>,
session_id: u64,
}
enum AppRequestCommandV0 {
Fetch(AppFetchContentV0),
Pin,
UnPin,
Delete,
Create,
FileGet,
FilePut,
}
enum AppFetchContentV0 {
Get, // without subscribing
Subscribe,
Update,
ReadQuery,
WriteQuery,
RdfDump,
History,
SignatureStatus,
SignatureRequest,
SignedSnapshotRequest,
}
enum AppRequestPayload {
V0(AppRequestPayloadV0),
}
enum AppRequestPayloadV0 {
Create(DocCreate),
Query(DocQuery),
Update(DocUpdate),
AddFile(DocAddFile),
Delete(DocDelete),
SmallFilePut(SmallFile),
/// content_type (IANA media type)
RandomAccessFilePut(String),
/// an empty Vec ends the upload
RandomAccessFilePutChunk((u32, Vec<u8>)),
}
struct NuriV0 {
identity: Option<UserId>,
target: NuriTargetV0,
entire_store: bool,
objects: Vec<ObjectRef>,
signature: Option<ObjectRef>,
branch: Option<TargetBranchV0>,
overlay: Option<OverlayLink>,
access: Vec<NgAccessV0>,
topic: Option<TopicId>,
locator: Option<Locator>,
}
enum NuriTargetV0 {
/// targets the whole DataSet of the user
UserSite,
PublicStore,
ProtectedStore,
PrivateStore,
AllDialogs,
/// shortname of a Dialog
Dialog(String),
AllGroups,
/// shortname of a Group
Group(String),
Repo(RepoId),
None,
}
enum TargetBranchV0 {
Chat,
Stream,
Comments,
BackLinks,
Context,
BranchId(BranchId),
/// named branch or commit
Named(String),
Commits(Vec<ObjectId>),
}
enum OverlayLink {
Outer(Digest),
InnerLink(InnerOverlayLink),
Inner(Digest),
Inherit,
Public(PubKey),
Global,
}
struct InnerOverlayLink {
/// overlay public key ID
id: StoreOverlay,
store_overlay_readcap: ReadCap,
}
enum NgAccessV0 {
ReadCap(ReadCap),
Token(Digest),
ExtRequest(Vec<u8>),
Key(BlockKey),
Inbox(PubKey),
}
enum Locator {
V0(LocatorV0),
}
type LocatorV0 = Vec<BrokerServer>;
struct BrokerServer {
content: BrokerServerContentV0,
/// peerId of the server
peer_id: PubKey,
/// optional signature over content by peer_id
sig: Option<Sig>,
}
struct BrokerServerContentV0 {
servers: Vec<BrokerServerTypeV0>,
version: u32,
}
enum BrokerServerTypeV0 {
/// optional port number, defaults to 1440
Localhost(u16),
BoxPrivate(Vec<BindAddress>),
Public(Vec<BindAddress>),
BoxPublicDyn(Vec<BindAddress>),
Domain(String),
}
-
NuriV0.identity : None for personal identity
-
NuriV0.entire_store : If it is a store, will include all the docs belonging to the store. not used otherwise
-
NuriV0.objects : used only for FileGet.
-
NuriV0.branch : if None, the main branch is chosen
-
InnerOverlayLink.store_overlay_readcap : The store has a special branch called
Overlay
that is used to manage access to the InnerOverlay. Only the ReadCapSecret is needed to access the InnerOverlay. The full readcap of this branch is needed in order to subscribe to the topic and decrypt the events, and hence be able to subscribe to refreshes of the InnerOverlay. The branchId can be found in the branch Definition. It can be useful to subscribe to this topic if the user is a member of the store’s repo, so it will be notified of BranchCapRefresh on the overlay. To the contrary, if the user is an external user to the store, they won’t be able to subscribe and they will loose access to the InnerOverlay after a BranchCapRefresh of the overlay branch of the store. -
BrokerServerTypeV0::Domain : accepts an optional trailing “:
port
” number
Response
enum AppResponse {
V0(AppResponseV0),
}
enum AppResponseV0 {
SessionStart(AppSessionStartResponse),
TabInfo(AppTabInfo),
State(AppState),
Patch(AppPatch),
History(AppHistory),
SignatureStatus(Vec<(String, Option<String>, bool)>),
Text(String),
FileUploading(u32),
FileUploaded(ObjectRef),
FileBinary(Vec<u8>),
FileMeta(FileMetaV0),
QueryResult(Vec<u8>),
Graph(Vec<u8>),
Ok,
True,
False,
Error(String),
EndOfStream,
Nuri(String),
}
enum AppSessionStartResponse {
V0(AppSessionStartResponseV0),
}
struct AppSessionStartResponseV0 {
private_store: RepoId,
protected_store: RepoId,
public_store: RepoId,
}
DocCreate
Creates a new Document.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Create,
nuri: NuriV0::new_empty(),
payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::Create(
DocCreate {
...
},
))),
}
struct DocCreate {
store: StoreRepo,
class: BranchCrdt,
destination: DocCreateDestination,
}
enum DocCreateDestination {
Store,
Stream,
MagicCarpet,
}
Response
replies with an AppResponseV0::Nuri(string)
containing the string representation of the Document’s Nuri, of the form did:ng:o:[repo_id]:v:[overlay_id]
.
DocUpdate
Updates the graph or discrete nature of the Document, or both.
Replied with AppResponseV0::Ok
.
If set, the NuriV0.branch
should be of the variant BranchId(_)
. If None, the main branch will be used.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::Update),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
... // all the rest empty
},
payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::Update(
DocUpdate {
...
},
))),
}
struct DocUpdate {
heads: Vec<ObjectId>,
graph: Option<GraphUpdate>,
discrete: Option<DiscreteUpdate>,
}
struct GraphUpdate {
// serialization of Vec<Triple>
inserts: Vec<u8>,
// serialization of Vec<Triple>
removes: Vec<u8>,
}
enum DiscreteUpdate {
/// A yrs::Update
YMap(Vec<u8>),
YArray(Vec<u8>),
YXml(Vec<u8>),
YText(Vec<u8>),
/// An automerge::Change.raw_bytes()
Automerge(Vec<u8>),
}
GraphUpdate
is not implemented for now.
WriteQuery
A SPARQL Update query. Can span multiple Documents (by indicating a GRAPH <Nuri>
or WITH <Nuri>
, and optional USING <Nuri>
with a Nuri of the form did:ng:o:v
).
The AppRequestV0.nuri
is mandatory and will represent the default graph.
Replied with a AppResponseV0::Ok
.
The NuriV0.target
cannot be a UserSite
, AllDialogs
nor AllGroups
.
The NuriV0.branch
cannot be a Commits(_)
.
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::WriteQuery),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
... // all the rest empty
},
payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::Query(
DocQuery::V0 {
...
},
))),
}
enum DocQuery {
V0 {
sparql: String,
base: Option<String>,
},
}
-
DocQuery::V0.base : an optional base to resolve all your relative URIs of resources in the SPARQL Update query.
-
DocQuery::V0.sparql : the text of your SPARQL Update
FilePut
Uploads a binary file into the repository (Document). The API is composed of several calls that should be made sequentially.
The first call must be an AppRequestPayloadV0::RandomAccessFilePut
that will return an upload_id.
Immediately followed by one or more AppRequestPayloadV0::RandomAccessFilePutChunk
.
And eventually finished with one call to AppRequestPayloadV0::AddFile
.
first Request : RandomAccessFilePut
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::FilePut,
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
... // all the rest empty
},
payload: Some(AppRequestPayload::V0(AppRequestPayloadV0::RandomAccessFilePut(
content_type // a string representing an IANA media type
))),
}
first Response
AppResponseV0::FileUploading(upload_id) // a u32
one or more chunk put Request(s) : RandomAccessFilePutChunk
upload your file in chunks of maximum 1 048 564 bytes, for best efficiency.
Repeat this call until you finished upload all your file.
Add another extra call at the end, with a size of zero. This will indicate that you are done with uploading chunks.
Replied with a AppResponseV0::Ok
for each chunk that is non-empty.
The last call with an empty chunk is replied with an AppResponseV0::FileUploaded(reference)
containing an ObjectRef
to the uploaded file, that you should use in the next call AppRequestPayloadV0::AddFile
.
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::FilePut,
nuri: NuriV0::new_empty(),
payload: Some(AppRequestPayload::V0(
AppRequestPayloadV0::RandomAccessFilePutChunk(
(upload_id, chunk) // chunk is a Vec<u8>
)
)),
}
- nuri : can be omitted in those calls, unlike the 2 other type of calls
last Request : AddFile
Finally you make an AppRequestPayloadV0::AddFile
call that will attach the Object to your document’s branch.
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::FilePut,
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: // can be omitted (defaults to main branch)
... // all the rest empty
},
payload: Some(AppRequestPayload::V0(
AppRequestPayloadV0::AddFile(
DocAddFile {
...
}
)
)),
}
struct DocAddFile {
filename: Option<String>,
object: ObjectRef,
}
-
DocAddFile.object : must be the reference you obtained from the last call to
RandomAccessFilePutChunk
. -
DocAddFile.filename : an optional filename. usually it is the original filename on the filesystem when selecting the binary file to upload.
final Response
Finally, you get an AppResponseV0::Ok
if everything went well.
FileGet
Downloads a binary file
The request nuri needs to have both a target
and one, and only one objects
.
Replied with a stream of AppResponseV0
.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::FileGet,
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: // can be omitted (defaults to main branch)
objects: [object_ref], // the requested object/file reference
... // all the rest empty
},
payload: None,
}
first Response : FileMeta
The first response in the stream is always an AppResponseV0::FileMeta(FileMetaV0)
.
struct FileMetaV0 {
content_type: String,
size: u64,
}
more Responses : FileBinary
Then a series of AppResponseV0::FileBinary(chunk)
containing all the chunks (Vec<u8>
) of the file, are send in the streamed response.
final response : EndOfStream
And eventually, once all the data has been transferred, an AppResponseV0::EndOfStream
is sent in the stream.
Subscribe
Subscribes to all the events/commits of a branch.
The response is streamed and long-lived, until cancelled.
The first response contains some meta-data about the branch called TabInfo
. It is used mostly by the App to display the document header and instantiate the document’s viewer.
Then a second response contains the materialized state of the document as it is known by the local Verifier at the moment of the subscribe request.
Later on, new updates are added to the stream as they arrive on the local replica (including from the changes made locally by the user).
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::Subscribe),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: // can be omitted (defaults to main branch)
... // all the rest empty
},
payload: None,
}
first Response : AppTabInfo
The first response is a AppResponseV0::TabInfo(AppTabInfo)
.
struct AppTabInfo {
branch: Option<AppTabBranchInfo>,
doc: Option<AppTabDocInfo>,
store: Option<AppTabStoreInfo>,
}
struct AppTabBranchInfo {
id: Option<String>,
readcap: Option<String>,
comment_branch: Option<String>, // not implemented
class: Option<String>,
}
struct AppTabDocInfo {
nuri: Option<String>,
is_store: Option<bool>,
is_member: Option<String>,
title: Option<String>,
icon: Option<String>,
description: Option<String>,
authors: Option<Vec<String>>, // not implemented
inbox: Option<String>, // not implemented
can_edit: Option<bool>,
}
struct AppTabStoreInfo {
repo: Option<StoreRepo>,
overlay: Option<String>,
has_outer: Option<String>, // not implemented
store_type: Option<String>,
readcap: Option<String>,
is_member: Option<String>,
inner: Option<String>,
title: Option<String>,
icon: Option<String>,
description: Option<String>,
}
second Response : AppState
The second response is a AppResponseV0::State(AppState)
.
struct AppState {
pub heads: Vec<ObjectId>,
pub head_keys: Vec<ObjectKey>,
pub graph: Option<GraphState>,
pub discrete: Option<DiscreteState>,
pub files: Vec<FileName>,
}
struct GraphState {
// serialization of Vec<Triple>
pub triples: Vec<u8>,
}
enum DiscreteState {
/// A yrs::Update
YMap(Vec<u8>),
YArray(Vec<u8>),
YXml(Vec<u8>),
YText(Vec<u8>),
// the output of Automerge::save()
Automerge(Vec<u8>),
}
struct FileName {
pub name: Option<String>,
pub reference: ObjectRef,
pub nuri: String,
}
followed by AppPatch
Then on every update done on the branch, a new AppResponseV0::Patch(AppPatch)
is sent in the stream.
struct AppPatch {
pub commit_id: String,
pub commit_info: CommitInfoJs,
// or graph, or discrete, or both, or other.
pub graph: Option<GraphPatch>,
pub discrete: Option<DiscretePatch>,
pub other: Option<OtherPatch>,
}
struct CommitInfoJs {
pub past: Vec<String>,
pub key: String,
pub signature: Option<String>,
pub author: String,
pub timestamp: String,
pub final_consistency: bool,
pub commit_type: CommitType,
pub branch: Option<String>,
pub x: u32,
pub y: u32,
}
enum CommitType {
TransactionGraph,
TransactionDiscrete,
TransactionBoth,
FileAdd,
FileRemove,
Snapshot,
Compact,
AsyncSignature,
SyncSignature,
Branch,
UpdateBranch,
BranchCapRefresh,
CapRefreshed,
Other,
}
struct GraphPatch {
// serialization of Vec<Triple>
pub inserts: Vec<u8>,
// serialization of Vec<Triple>
pub removes: Vec<u8>,
}
enum DiscretePatch {
/// A yrs::Update
YMap(Vec<u8>),
YArray(Vec<u8>),
YXml(Vec<u8>),
YText(Vec<u8>),
/// An automerge::Change.raw_bytes() or a concatenation of several of them.
Automerge(Vec<u8>),
}
enum OtherPatch {
FileAdd(FileName),
FileRemove(ObjectId),
AsyncSignature((String, Vec<String>)),
Snapshot(ObjectRef),
Compact(ObjectRef),
Other,
}
ReadQuery
A SPARQL query (read-only).
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::ReadQuery),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: // not implemented yet
... // all the rest empty
},
payload: Some(AppRequestPayload::V0(
AppRequestPayloadV0::Query(
DocQuery::V0 {
...
}
)
)),
}
enum DocQuery {
V0 {
sparql: String,
base: Option<String>,
},
}
-
AppRequestV0.nuri.target : represents the default graph. can be NuriV0::UserSite or NuriV0::None and in those cases, the union graph of all the graphs is used as default graph.
-
DocQuery::V0.base : an optional base to resolve all your relative URIs of resources in the SPARQL Query.
-
DocQuery::V0.sparql : the text of your SPARQL Query
Response
Depending on the type of query, the response differs.
-
for SELECT queries: an
AppResponseV0::QueryResult(buffer)
where buffer is aVec<u8>
containing a UTF-8 serialization of a JSON string representing a JSON Sparql Query Result. see SPARQL Query Results JSON Format. -
for CONSTRUCT an
AppResponseV0::Graph(buffer)
where buffer is aVec<u8>
containing a BARE serialization of aVec<Triple>
. -
for ASK queries:
AppResponseV0::True
orAppResponseV0::False
RdfDump
Gets the full dump of all the quads contained in the UserSite (whole local DataSet of the User).
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::RdfDump),
nuri: NuriV0::new_empty()
payload: None,
}
Response
AppResponseV0::Text(string)
string
being a Turtle output of all the quads.
History
Fetches the current (HEADs) history of commits for this branch.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::History),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
... // all the rest empty
},
payload: None
}
Response
The response is a AppResponseV0::History(AppHistory)
struct AppHistory {
pub history: Vec<(ObjectId, CommitInfo)>,
pub swimlane_state: Vec<Option<ObjectId>>,
}
struct CommitInfo {
pub past: Vec<ObjectId>,
pub key: ObjectKey,
pub signature: Option<ObjectRef>,
pub author: String,
pub timestamp: Timestamp,
pub final_consistency: bool,
pub commit_type: CommitType,
pub branch: Option<ObjectId>,
pub x: u32,
pub y: u32,
}
-
AppHistory.history : in order, with the newest commits first. The first part of the tuple (ObjectId) is the CommitID.
-
AppHistory.swimlane_state : can be discarded. it is only used by our GUI representation of the history in the Apps. Same with the x and y values in CommitInfo.
SignatureStatus
Fetches the Signature status of the branch, at the HEADs.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::SignatureStatus),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
... // all the rest empty
},
payload: None
}
Response
The response is a AppResponseV0::SignatureStatus(Vec<(String, Option<String>, bool)>)
Which is a list of commits at the HEAD. Each commit is represented by a tuple that contains :
- the commit ID printed as a string (44 characters)
- an optional string that is present only if the commit is a signature. in this case, the string represents a list of the signed commits
c:[commit_id]:k:[commit_key]
joined with:
(at least one commit is present), followed by partial Nuri for the signature object:s:[signature_object_id]:k:[signature_object_key]
. - a boolean that indicates if the commit is a snapshot. (in this case, only one commit is present in the HEADs)
SignatureRequest
Requests that the HEADs commit(s) be asynchronously signed.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::SignatureRequest),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
... // all the rest empty
},
payload: None
}
Response
If the signature is immediately available, the response is AppResponseV0::True
. The new HEAD containing the signature can be immediately fetched with a call to SignatureStatus
.
If the signature is not immediately available (because the quorum needs to sign it and it can take some time), the response is AppResponseV0::False
. A notification will be sent to the user once the signature is ready (not implemented yet).
SignedSnapshotRequest
Requests that a snapshot be taken from the current materialized state at the HEADs, and that this new commit containing the snapshot, be signed asynchronously.
Request
// example AppRequestV0
AppRequestV0 {
...,
command : AppRequestCommandV0::Fetch(AppFetchContentV0::SignedSnapshotRequest),
nuri: NuriV0 {
target: NuriTargetV0::Repo(repo_id),
overlay: Some(overlay_id),
branch: //not implemented. defaults to main branch
... // all the rest empty
},
payload: None
}
Response
If the signature is immediately available, the response is AppResponseV0::True
. The newly created snapshot and its signature can be immediately fetched with a call to SignatureStatus
.
If the signature is not immediately available (because the quorum needs to sign it and it can take some time), the response is AppResponseV0::False
. A notification will be sent to the user once the signature is ready (not implemented yet). The snapshot is already available as the last commit in HEAD, but it isn’t signed yet.