Distributed Database
Maturity: Draft — not yet validated by a reference implementation. Subject to change during Runtime development.
See Protocol Specifications for reading order and the full specification index.
Abstract
This specification defines the Faberio Distributed Database service — the platform capability for storing, querying, and replicating structured application data across nodes within an information space.
Applications access data through a standardized database API, the Runtime HTTP API, or — when supported by the storage engine — through native database functions. The Runtime handles replication, consistency, and synchronization transparently, using Messaging and RPC internally to coordinate state between nodes.
1. Scope
This specification defines:
- database API for applications
- HTTP API for external applications
- native engine access for change-stream-capable databases
- record model and identifiers
- storage engine abstraction
- replication architecture and flow
- consistency and conflict resolution (CRDT merge)
- local-first execution
- information space scoping
- export, backup, and portability
- access control integration
Future protocol work (planned, not required in the reference MVP) includes flexible full synchronization — lazy loading, paginated snapshots, and metadata-first replication. See section 9.4.
This specification does not define:
- messaging protocol (see Messaging)
- RPC protocol (see RPC)
- object identifier format (see Object Identifier)
- identity and access control model (see Identity Model, Data Access Control)
- file storage (see Distributed File Storage)
- specific SQL dialect or query language
2. Motivation
Distributed applications require structured data that remains available across independently operated nodes. Building replication, consistency, and conflict resolution into every application is complex and error-prone.
Faberio provides a distributed database as a platform service. Applications read and write data as if using a local database, while the Runtime replicates changes across participating nodes within an information space.
The database layer is separated from the replication transport layer. Application data lives in persistent storage engines; replication coordination uses ephemeral Messaging and RPC on the control plane.
3. Concepts
3.1 Record
A record is a single data object stored in the database. Every record has a UUIDv7 identifier assigned at creation time as defined in the Object Identifier specification.
3.2 Collection
A collection (or table) is a named group of records with a shared schema. Collections are defined by applications. The Runtime replicates all records within authorized information spaces regardless of collection.
3.3 Storage Engine
A storage engine is the underlying database implementation used by the Runtime on each node. Faberio does not mandate a specific engine — implementations MAY use embedded databases, SQL databases, document stores, or key-value engines.
3.4 Replica
A replica is a copy of a record maintained on a participating node. Each node in an information space holds local replicas of replicated records.
4. Database API
4.1 Application Interface
Applications interact with the distributed database through one of the integration modes described in section 4.4. The standardized database API provides:
| Operation | Description |
|---|---|
create | Insert a new record with a UUIDv7 identifier |
read | Retrieve a record by UUIDv7 identifier |
update | Modify an existing record |
delete | Remove a record |
query | Search records within a collection |
The same operations are available through the HTTP API for external applications. Applications MUST NOT implement replication, node coordination, or conflict resolution. These are handled by the Runtime.
4.2 Record Format
Every record MUST contain:
| Field | Type | Description |
|---|---|---|
id | UUIDv7 | Primary identifier, assigned at creation |
collection | string | Collection (table) name |
data | object | Application-defined fields |
version | integer | Replication version, managed by the Runtime |
updated_at | timestamp | Last modification time (Unix milliseconds) |
The id field is immutable. The version field is incremented by the Runtime on every update and is used for concurrent update detection.
4.3 Local-First Execution
All database operations execute against the local replica on the current node. The Runtime synchronizes changes to remote nodes asynchronously.
This enables:
- low-latency reads and writes
- offline operation
- continued availability during network partitions
When connectivity is restored, the Runtime synchronizes pending changes across nodes.
4.4 Application Integration
Applications MAY interact with the distributed database in three ways. In all modes, the Runtime owns replication.
| Mode | Application location | Access method |
|---|---|---|
| In-runtime SDK | Inside the Runtime | Faberio SDK database API |
| HTTP API | Outside the Runtime | Runtime HTTP endpoints |
| Native engine access | Inside or outside the Runtime | Native database client (change-stream engines only) |
In-Runtime SDK
The default integration mode. Applications run inside the Runtime and use the Faberio SDK to access the database API described in section 4.1.
HTTP API
Applications that run outside the Runtime — on separate servers, containers, or devices — access the distributed database through the Runtime's HTTP API.
The HTTP API exposes the same operations as the SDK (create, read, update, delete, query) as REST endpoints. The Runtime enforces authentication, information space scoping, and access control on every request.
POST /api/v1/database/records — create
GET /api/v1/database/records/{id} — read
PUT /api/v1/database/records/{id} — update
DELETE /api/v1/database/records/{id} — delete
GET /api/v1/database/records — query
External applications MUST NOT bypass the HTTP API to access the storage engine directly, unless native engine access is explicitly enabled.
Native Engine Access
For storage engines that support change streams, the Runtime MAY expose native database access. See section 5.2 for details.
In all integration modes, applications MUST NOT implement replication, node coordination, or conflict resolution.
5. Storage Engine Abstraction
The Runtime separates replication from the storage backend. Applications use the same database API through the SDK or HTTP API regardless of the underlying engine.
Supported engine types include:
| Type | Examples |
|---|---|
| Embedded | SQLite, LevelDB, RocksDB |
| SQL | PostgreSQL, MySQL |
| Document | MongoDB-compatible stores |
| Key-value | Redis-compatible stores |
The Runtime synchronizes data independently of the storage engine. Replication operates on the record abstraction, or on engine change events when native access is used.
5.1 SDK and HTTP API Access
In SDK and HTTP API modes, applications interact through the Faberio database API described in section 4.1 — either via the SDK (in-runtime) or HTTP endpoints (external). The Runtime wraps the underlying storage engine and normalizes all operations into the record model.
Change detection is performed by the Runtime through engine hooks, triggers, or polling, depending on the storage engine capabilities.
5.2 Native Engine Access
For storage engines that support change streams, the Runtime MAY expose native database access to applications. In this mode, applications interact with the database using the engine's native client, driver, or query functions instead of the Faberio database API.
Supported Capability
Native engine access requires the storage engine to provide a change stream or equivalent change notification mechanism that allows the Runtime to observe all modifications in real time.
| Engine | Change stream mechanism |
|---|---|
| MongoDB | Change Streams |
| PostgreSQL | Logical replication / NOTIFY (implementation-dependent) |
Engines without change stream support MUST use SDK or HTTP API access.
Application Experience
In native access mode, the Runtime provides applications with a native database connection scoped to the authorized information space. Applications MAY run inside or outside the Runtime. For example, a MongoDB runtime MAY expose a native MongoDB client:
// Application uses native MongoDB API
const orders = db.collection('orders');
await orders.insertOne({ _id: uuidv7(), total: 150 });
Applications use the full query capabilities of the underlying engine — aggregation pipelines, indexes, transactions, and other engine-specific features — without going through the Faberio record abstraction.
Runtime Responsibilities
Even in native access mode, the Runtime retains full responsibility for replication:
- Subscribes to the engine's change stream
- Translates change events into replication notifications
- Coordinates synchronization via Messaging and RPC
- Enforces information space scoping and access control
Applications write to the native database directly, but replication remains entirely runtime-internal.
Requirements for Native Access
When native engine access is enabled, implementations MUST:
- ensure all replicated documents include a UUIDv7 identifier as defined in the Object Identifier specification
- subscribe to the engine change stream for all collections within the authorized information space
- enforce access control before granting native database connections
- scope native connections to a single information space
Applications MUST NOT:
- configure or manage change stream subscriptions
- bypass information space scoping through native connections
- disable or interfere with runtime-managed replication
6. Replication Architecture
6.1 Overview
Replication synchronizes database records across participating nodes within an information space. Applications write to the local replica; the Runtime propagates changes to remote nodes.
6.2 Data Plane vs Control Plane
| Layer | Component | Role |
|---|---|---|
| Data plane | Local storage engine | Persistent record storage |
| Control plane | Messaging | Async change notifications |
| Control plane | RPC | Sync state fetch and CRDT merge |
Application data is never transmitted as messaging payloads at scale. Messaging notifies remote nodes that a record has changed; RPC fetches the actual state when needed. The data plane holds the authoritative record content.
6.3 Replication Flow
- Application updates a record through the database API.
- Runtime writes to the local replica and increments
version. - Replication engine sends a messaging notification to peers.
- Remote replication engine receives the notification.
- Remote node issues an RPC call to fetch the current record state.
- Remote node applies the update to its local replica.
- Application on the remote node reads the updated data through the database API.
6.4 Change Detection
The Runtime MUST detect all changes to replicated records. The detection mechanism depends on the integration mode and storage engine:
| Integration mode | Change detection |
|---|---|
| SDK / HTTP API | Runtime hooks, triggers, or polling on the record abstraction |
| Native engine access | Engine change stream subscription |
In both cases, the Runtime MUST capture:
- record creation
- record update
- record deletion
For native engine access, the Runtime subscribes to the engine's change stream (e.g. MongoDB Change Streams) and translates change events into replication operations. Applications are not aware of change stream subscriptions.
Detected changes trigger the replication flow described in section 6.3.
6.5 Replication Scope
Replication is scoped to an information space. Records belong to the information space configured for the application. Changes are propagated only to nodes participating in the same information space.
Records MUST NOT be replicated across information space boundaries.
When replication operates in user-level mode, the Runtime synchronizes only records the user is authorized to access during background replication. See Data Access Control for replication authorization contexts.
7. Consistency Model
7.1 Eventual Consistency
The distributed database provides eventual consistency by default. All replicas converge to the same state given sufficient time and network connectivity.
Applications MUST be designed to tolerate temporary inconsistency between nodes. Reads immediately after a write on a remote node MAY return stale data until replication completes.
7.2 Version Tracking
Every record carries a version integer managed by the Runtime. The version is incremented on every update and is used for:
- detecting concurrent modifications
- ordering replication and sync operations
- identifying when CRDT merge is required during synchronization
7.3 Read Consistency
| Read type | Behavior |
|---|---|
| Local read | Returns the local replica immediately (may be stale relative to other nodes) |
| Consistent read | Runtime MAY wait for replication acknowledgment before returning (optional, application-configured) |
By default, all reads are local reads for maximum performance and offline availability.
8. Conflict Resolution
Faberio resolves concurrent updates with CRDT merge — deterministic, commutative merge rules that converge all replicas to the same state without a central coordinator. Applications MUST NOT implement their own conflict resolution for replicated records.
8.1 Concurrent Update Detection
A concurrent update occurs when two nodes modify the same record while disconnected or before replication completes, producing divergent field states at the same logical object id.
The Replication engine detects concurrent modifications during synchronization using version comparison and CRDT metadata via platform.replication.compare RPC calls.
8.2 CRDT Merge (Default)
The default resolution strategy is CRDT merge. When concurrent updates are detected, the Runtime merges replicas using the CRDT type declared for each field (or the collection default).
| CRDT type | Typical use | Merge behaviour |
|---|---|---|
| Register | Scalar fields (string, number, boolean) | Converges to a single value using causal metadata (default for undeclared fields) |
| Counter | Counters, quantities | Sum of independent increments |
| OR-Set | Tags, lists without ordering | Set union with tombstones for removals |
| Map | Nested objects | Recursive merge of child CRDTs |
| Text | Collaborative text fields | Character-wise CRDT merge (optional; future) |
Applications MAY declare CRDT types per collection field in schema metadata. Collections MAY specify a default CRDT type for unspecified fields (Register if omitted).
| Strategy | Description |
|---|---|
| CRDT merge | Automatic deterministic merge (default) |
| Manual | Merge cannot proceed automatically; flagged for application or operator review via platform.replication.resolve |
Implementations MUST apply CRDT merge unless the collection or operator configures Manual for that resource.
8.3 Merge Flow
9. Offline Operation
9.1 Local Writes
Applications MAY read and write the local replica while disconnected from remote nodes. All changes are stored locally and tracked by the Replication engine.
9.2 Synchronization on Reconnect
When connectivity is restored, the Runtime selects a synchronization strategy based on how long the node was offline.
| Offline duration | Strategy | Description |
|---|---|---|
| Short | Incremental | Replay pending changes and fetch missed notifications via platform.replication.fetch |
| Extended | Full state | Request complete record set via platform.replication.sync and platform.replication.snapshot |
Incremental Synchronization
Incremental synchronization is the default for short offline periods. The Runtime:
- sends messaging notifications for all pending local changes
- fetches missed remote changes via
platform.replication.fetchRPC - detects concurrent updates and merges replicas via CRDT
- converges all replicas to a consistent state
Full State Synchronization
When a node has been offline for an extended period, incremental synchronization is insufficient. Change notifications sent while the node was unreachable are not retained — see Messaging section 8.3.
In this case, the node requests a full copy of all database records from a peer instead of synchronizing individual change messages:
- the reconnecting node sends
platform.replication.syncvia Messaging - the peer transfers all records via
platform.replication.snapshotRPC - the reconnecting node applies the received state to its local replica
- the Runtime replays local pending offline changes and merges concurrent updates via CRDT
- incremental synchronization resumes for subsequent changes
A node SHOULD use full state synchronization when:
| Trigger | Description |
|---|---|
| Extended offline | Node was unreachable longer than a configurable threshold (recommended default: 24 hours) |
| First join | Node has no local replica for the information space |
| Large divergence | Local state is significantly behind peers |
| Operator request | Manual resync initiated by the node operator |
When replication operates in user-level mode, the snapshot MUST include only records the user is authorized to access. See Data Access Control.
Full state transfer MAY be split into multiple chunked RPC responses. The Runtime MUST verify record version integrity after transfer completes.
9.3 Pending Changes
The Runtime MUST maintain a local log of changes made while offline. This log is local to each node and is not replicated — it tracks which local records need synchronization.
9.4 Future: Flexible Full Synchronization
The baseline model in sections 9.2–9.3 transfers complete record sets during extended offline recovery and first join. That behaviour is sufficient for small information spaces and server-node validation, but does not scale to large datasets, mobile devices with limited storage, or user-level replicas that need only a subset of records.
Future protocol and Runtime versions SHOULD implement flexible full synchronization — multiple strategies that converge replicas without always downloading every record and every column up front.
| Mechanism | Description | Use case |
|---|---|---|
| Lazy loading | The Runtime replicates or reconstructs an index locally (identifiers, versions, collection membership) and fetches the full record payload from a source peer only when the application reads the record or the Runtime prefetches explicitly | Large histories; user devices that browse infrequently used data |
| Paginated snapshot | platform.replication.snapshot accepts cursor, limit, and collection filters; the reconnecting node applies snapshot pages sequentially until the session completes | Extended offline recovery without single huge RPC responses |
| Metadata-first replication | Replicate record headers first — UUIDv7, version, collection key, scalar fields, and presence flags for large columns — and defer blob or large column values until an on-demand fetch | Community Hub with documents; business apps with document fields; media attachments |
Behaviour requirements (planned)
Future implementations that support these mechanisms MUST:
- apply source-side authorization on every snapshot page and on-demand fetch (see Data Access Control — Source-Side Authorization)
- preserve local-first reads — applications read the local replica; the Runtime triggers background fetch when data is not yet local
- expose sync progress to operators (pages completed, bytes remaining, deferred objects count)
- mark locally incomplete records explicitly (for example
payload_status: local|deferred|fetching) so applications can show placeholders instead of stale data - fetch deferred payload through
platform.replication.fetch(or a paginated snapshot continuation) with the same user or node authorization context as full replication - resume incremental synchronization after metadata-first or paginated full sync completes
Applications MAY declare a sync profile per collection (for example full, metadata_first, lazy) where the Runtime supports it. Operators MAY set defaults per information space. User-level replication SHOULD default to metadata-first or lazy profiles for large collections on device nodes.
This section describes planned protocol capability. The reference MVP MAY implement only monolithic full snapshot and incremental sync as defined in sections 9.2–9.3.
See also Distributed File Storage — Flexible full synchronization for the file-object equivalent (metadata without content bytes).
10. Export, Backup, and Portability
Data decoupling requires that operators and authorized users can leave with their data — without a vendor export gate. Faberio supports backup, portable export, and migration at the Runtime layer while preserving information space boundaries and Data Access Control.
10.1 Operator Backup
An operator backup is a local snapshot of everything required to restore a node’s contribution to an information space after hardware failure or redeployment.
| Component | Included in node backup |
|---|---|
| Database replica | All records the node holds for authorized information spaces |
| Platform metadata | Node membership records, access policies, identity records (public keys only) |
| Pending sync state | Local pending change log (section 9.3) |
| Node private key | Stored outside the replicated database in the operator’s secret store — MUST be backed up separately |
Backups MUST NOT include user private keys unless the node is an active user device that legitimately holds them.
Implementations SHOULD support:
- Storage-engine backup — native dump/snapshot of the embedded database when the engine provides it.
- Runtime-coordinated backup — quiesce or snapshot the local replica through operator tooling while replication is paused or from a read-consistent view.
Backups are operator-controlled. The Runtime MUST NOT upload backups to a central Faberio service by default.
10.2 Portable Export
A portable export is an open, self-contained archive of database records (and, when file storage is enabled, file objects — see Distributed File Storage §10) that an authorized operator or application may produce for compliance, archival, or migration.
| Requirement | Rule |
|---|---|
| Authorization | Export MUST enforce source-side authorization — only records the exporter is permitted to read are included |
| Scope | Export is scoped to one information_space_id; cross-space export in a single archive is forbidden |
| Identifiers | Records MUST retain UUIDv7 id values and collection keys so imports can preserve object identity |
| Format | Implementations SHOULD use a documented open format — for example JSON Lines for record payloads plus a manifest (information_space_id, export timestamp, schema/collection list, record count) |
| Integrity | Manifest SHOULD include per-record version and optional content hashes for verification on import |
Portable export MAY be exposed through operator CLI, HTTP API (admin-authenticated), or SDK hooks. Applications MUST NOT bypass DAC when exporting user-visible data.
10.3 Migrate Node (Same Information Space)
Node migration moves a Runtime deployment to new infrastructure while the node remains a member of the same information space.
Typical flow:
- Operator provisions a new host (NixOS module, container, or PC/laptop).
- Operator restores a backup (section 10.1) or joins the new deployment as an approved member and receives state via
platform.replication.snapshot(RPC). - Operator updates connectivity endpoints (addresses, TLS certificates) in local config and discovery/bootstrap settings (Node Discovery §10.4).
- Old deployment is decommissioned after peers converge; membership record UUIDv7
idfor the node MAY remain unchanged if the same node record and keys are preserved.
If the operator rotates the node key pair, the Node ID changes but the node record UUIDv7 identifier SHOULD remain the same (see Identity Model — Node key rotation). Peers MUST accept the updated public key after membership is still approved.
10.4 Import and Space Fork
Import creates or updates a local replica from a portable export. Import MUST:
- verify manifest
information_space_idmatches the target space (or a new space created for import) - re-apply access policies from the archive or reconcile with existing policies on the target node
- merge imported records via normal replication rules (CRDT merge for conflicts)
Space fork — copying data into a new information space with a new information_space_id — is an operator workflow: export from space A, create space B, import, then re-approve nodes and users in space B. Faberio does not merge two live information spaces automatically; fork preserves data custody while establishing a new trust boundary.
The reference MVP MAY ship operator backup and peer snapshot recovery first; full portable export/import tooling is expected in later hardening.
11. Information Space Scoping
Database records are scoped to an information space. The Runtime MUST verify application authorization for the information space before allowing read or write operations, as defined in the Node Discovery specification.
Replication propagates changes only within the authorized information space. A record created in space A is never replicated to nodes in space B.
12. Access Control
Access to database records is governed by the Data Access Control specification. Before reading or writing replicated data, the Runtime verifies the user's cryptographic signature, approval status, and access policies for the target resource scope (collection, column, or row).
Access control is enforced locally on each node. Remote replication does not bypass access policies — receiving nodes apply the same rules before making replicated data available to applications.
13. Security Requirements
Implementations MUST:
- assign UUIDv7 identifiers to all records at creation
- enforce information space scoping on all database operations
- enforce access control before granting read or write access
- encrypt data in transit during replication (via transport layer)
- sign all replication control messages (messaging and RPC)
Implementations MUST NOT:
- expose replication internals to applications
- replicate records across information space boundaries
- grant access to records without evaluating access control policies
- allow applications to manipulate
versionor replication metadata directly
14. Examples
14.1 Record Creation
1. Application calls create({ collection: "orders", data: { total: 150 } })
2. Runtime assigns UUIDv7: 0195a3f0-9100-7898-bcde-000000000011
3. Record written to local replica (version: 1)
4. Replication engine notifies peers via messaging
5. Remote nodes fetch and store the record locally
14.2 Local-First Read
1. Application calls read("0195a3f0-9100-7898-bcde-000000000011")
2. Runtime returns local replica immediately
3. No network call required
14.3 Offline Update and Sync
1. Node A is offline
2. Application updates record (version: 2 → 3)
3. Change stored in local pending log
4. Node A reconnects
5. Runtime sends messaging notifications for pending changes
6. Remote nodes fetch updates via RPC
7. All replicas converge to version 3
14.4 Concurrent Update Merge
Node A: update record field status = "shipped" (version 2)
Node B: update record field notes = "fragile" (version 2)
1. Both nodes send messaging notifications
2. Replication engines detect concurrent updates via RPC compare
3. CRDT merge: Map merges both field updates — status and notes both preserved
4. Both nodes converge to the same merged record
14.5 Native MongoDB Access
1. Runtime configures MongoDB with change streams enabled
2. Application receives native MongoDB client scoped to information space
3. Application inserts document using native API:
db.orders.insertOne({ _id: "0195a3f0-9100-7898-bcde-000000000011", total: 150 })
4. Runtime change stream listener detects insert
5. Replication engine sends messaging notification to peers
6. Remote nodes fetch document via RPC and apply to local MongoDB
7. Application on remote node queries via native API:
db.orders.findOne({ _id: "0195a3f0-9100-7898-bcde-000000000011" })
14.6 Operator Portable Export
1. Administrator requests export for information space 0195a3f0-8000-7890-abcd-000000000001
2. Runtime evaluates DAC — includes only authorized collections and records
3. Export manifest written:
information_space_id: 0195a3f0-8000-7890-abcd-000000000001
exported_at: 1718000000000
collections: [ orders, messages ]
record_count: 1240
4. Record payloads written as JSON Lines (UUIDv7 id + version preserved)
5. Operator stores archive offline or imports into a forked space B after creating new membership