Distributed File Storage
Maturity: Draft — not yet validated by a reference implementation. Subject to change during Runtime development.
See Protocol Specifications for reading order and the full specification index.
Abstract
This specification defines the Faberio Distributed File Storage service — the platform capability for storing, retrieving, and replicating files across nodes within an information space.
Applications access files through a standardized file storage API, the Runtime HTTP API, or — when using a local filesystem backend — through native filesystem access. The Runtime handles replication, distribution, and synchronization transparently, using Messaging and RPC internally to coordinate file state between nodes.
1. Scope
This specification defines:
- file storage API for applications
- HTTP API for external applications
- native filesystem access for local filesystem backends
- file object model and identifiers
- storage backend abstraction
- replication architecture and flow
- large file handling and chunking
- consistency and conflict resolution (CRDT merge)
- local-first caching
- information space scoping
- export, backup, and portability
- access control integration
Future protocol work (planned, not required in the reference MVP) includes flexible full synchronization — lazy content loading, paginated snapshots, and metadata-first file replication. See section 9.4.
This specification does not define:
- messaging protocol (see Messaging)
- RPC protocol (see RPC)
- object identifier format (see Object Identifier)
- identity and access control model (see Identity Model, Data Access Control)
- structured data storage (see Distributed Database)
- specific filesystem or object store implementation
2. Motivation
Distributed applications require file storage that remains available across independently operated nodes. Files — documents, media, backups, binary assets — must be replicated without requiring applications to implement distribution logic.
Faberio provides distributed file storage as a platform service. Applications store and retrieve files as if using local storage, while the Runtime replicates file content and metadata across participating nodes within an information space.
Like the Distributed Database, the file storage layer is separated from the replication transport layer. File content lives in persistent storage backends; replication coordination uses ephemeral Messaging and RPC on the control plane.
3. Concepts
3.1 File Object
A file object is a stored file managed by the platform. Every file object has a UUIDv7 identifier assigned at creation time, independent of its path or filename.
3.2 File Path
A path is an optional logical location for a file object (e.g. /documents/report.pdf). Paths are application-defined and MAY change without affecting the file object's UUIDv7 identifier.
3.3 Storage Backend
A storage backend is the underlying implementation used to persist file content on each node. Implementations MAY use local filesystems, object stores, or embedded blob storage.
3.4 Replica
A replica is a copy of a file object's content and metadata maintained on a participating node.
4. File Storage API
4.1 Application Interface
Applications interact with distributed file storage through one of the integration modes described in section 4.4. The standardized file storage API provides:
| Operation | Description |
|---|---|
store | Store a new file, returns UUIDv7 identifier |
read | Retrieve file content by UUIDv7 identifier |
update | Replace file content for an existing identifier |
delete | Remove a file object |
list | List file objects by path prefix or metadata filter |
metadata | Retrieve file metadata without content |
The same operations are available through the HTTP API for external applications. Applications MUST NOT implement file replication, node coordination, or conflict resolution. These are handled by the Runtime.
4.2 File Object Format
Every file object MUST contain the following metadata:
| Field | Type | Description |
|---|---|---|
id | UUIDv7 | Primary identifier, assigned at creation |
path | string | Optional logical path |
size | integer | File size in bytes |
content_hash | string | SHA-256 hash of file content |
mime_type | string | MIME type of the content |
version | integer | Replication version, managed by the Runtime |
updated_at | timestamp | Last modification time (Unix milliseconds) |
The id field is immutable. File content is addressed by id, not by path. The content_hash enables integrity verification during replication.
4.3 Local-First Caching
All file operations execute against the local replica on the current node. The Runtime synchronizes file content to remote nodes asynchronously.
This enables:
- low-latency file access from local storage
- offline read and write operations
- continued availability during network partitions
When connectivity is restored, the Runtime synchronizes pending file changes across nodes.
4.4 Application Integration
Applications MAY interact with distributed file storage in three ways. In all modes, the Runtime owns replication.
| Mode | Application location | Access method |
|---|---|---|
| In-runtime SDK | Inside the Runtime | Faberio SDK file storage API |
| HTTP API | Outside the Runtime | Runtime HTTP endpoints |
| Native filesystem access | Inside or outside the Runtime | Native filesystem operations (local filesystem backends only) |
In-Runtime SDK
The default integration mode. Applications run inside the Runtime and use the Faberio SDK to access the file storage API described in section 4.1.
HTTP API
Applications that run outside the Runtime access distributed file storage through the Runtime's HTTP API.
The HTTP API exposes the same operations as the SDK (store, read, update, delete, list, metadata) as REST endpoints. The Runtime enforces authentication, information space scoping, and access control on every request.
POST /api/v1/files — store
GET /api/v1/files/{id} — read
PUT /api/v1/files/{id} — update
DELETE /api/v1/files/{id} — delete
GET /api/v1/files — list
GET /api/v1/files/{id}/metadata — metadata
External applications MUST access file storage through the HTTP API, unless native filesystem access is explicitly enabled.
Native Filesystem Access
For local filesystem backends, the Runtime MAY expose native filesystem access. In this mode, applications read and write files using standard filesystem operations instead of the Faberio file storage API. See section 5.2 for details.
5. Storage Backend Abstraction
The Runtime separates replication from the storage backend. Applications use the same file storage API through the SDK or HTTP API regardless of the underlying implementation.
Supported backend types include:
| Type | Examples |
|---|---|
| Local filesystem | POSIX filesystem, local directories |
| Object store | S3-compatible storage, MinIO |
| Embedded blob | SQLite BLOB, LevelDB |
The Runtime synchronizes file objects independently of the storage backend. Replication operates on the file object abstraction, or on filesystem change events when native access is used.
5.1 SDK and HTTP API Access
In SDK and HTTP API modes, applications interact through the Faberio file storage API described in section 4.1 — either via the SDK (in-runtime) or HTTP endpoints (external). The Runtime wraps the underlying storage backend and normalizes all operations into the file object model.
Change detection is performed by the Runtime through storage backend hooks, post-operation callbacks, or polling, depending on the backend capabilities.
5.2 Native Filesystem Access
For local filesystem backends, the Runtime MAY expose native filesystem access to applications. In this mode, applications interact with files using standard filesystem operations — open, read, write, rename, unlink — instead of the Faberio file storage API.
Supported Capability
Native filesystem access requires a local filesystem backend that supports filesystem change notification. The Runtime observes modifications in real time using platform-specific watch mechanisms:
| Platform | Change notification mechanism |
|---|---|
| Linux | inotify, fanotify |
| Cross-platform | fsnotify (wraps inotify, FSEvents, ReadDirectoryChangesW) |
Backends that do not support filesystem change notification — such as object stores or embedded blob storage — MUST use SDK or HTTP API access.
Application Experience
In native access mode, the Runtime provides applications with a filesystem directory scoped to the authorized information space. Applications MAY run inside or outside the Runtime. For example:
# Application writes directly to the filesystem
echo "report content" > /data/spaces/orders/documents/report.pdf
Applications use the full capabilities of the underlying filesystem — directory structures, file permissions, symbolic links, and other POSIX operations — without going through the Faberio file object abstraction.
The Runtime maps filesystem paths to file objects internally. When a new file appears in a watched directory, the Runtime assigns a UUIDv7 identifier and begins tracking it for replication.
Runtime Responsibilities
Even in native access mode, the Runtime retains full responsibility for replication:
- Watches authorized directories using inotify, fanotify, or fsnotify
- Detects file creation, modification, and deletion events
- Assigns UUIDv7 identifiers and computes
content_hashfor new files - Translates filesystem events into replication notifications
- Coordinates synchronization via Messaging and RPC
- Enforces information space scoping and access control
Applications write to the filesystem directly, but replication remains entirely runtime-internal.
Requirements for Native Access
When native filesystem access is enabled, implementations MUST:
- watch all directories within the authorized information space using inotify, fanotify, or fsnotify
- assign UUIDv7 identifiers to all new files as defined in the Object Identifier specification
- compute and store
content_hash(SHA-256) for all file content - enforce access control before granting filesystem directory access
- scope native directories to a single information space
Applications MUST NOT:
- configure or manage filesystem watch subscriptions
- bypass information space scoping through native directory access
- manipulate replication metadata (
version,content_hash) directly
6. Replication Architecture
6.1 Overview
File replication synchronizes file objects across participating nodes within an information space. Applications write to the local replica; the Runtime propagates changes to remote nodes.
6.2 Data Plane vs Control Plane
| Layer | Component | Role |
|---|---|---|
| Data plane | Local storage backend | Persistent file content and metadata |
| Control plane | Messaging | Async file change notifications |
| Control plane | RPC | Sync file fetch, chunk transfer, CRDT merge |
Messaging notifies remote nodes that a file has changed. RPC transfers file metadata and content (or chunks) when needed. File content is not embedded in messaging payloads.
6.3 Replication Flow
- Application stores a file through the file storage API.
- Runtime writes content to the local storage backend and assigns UUIDv7.
- Replication engine sends a messaging notification to peers.
- Remote replication engine receives the notification.
- Remote node fetches file metadata via RPC and verifies
content_hash. - Remote node fetches file content (or chunks) via RPC.
- Remote node writes the file to its local storage backend.
- Application on the remote node reads the file through the file storage API.
6.4 Change Detection
The Runtime MUST detect all changes to file objects:
- file creation
- content update
- file deletion
The detection mechanism depends on the integration mode and storage backend:
| Integration mode | Change detection |
|---|---|
| SDK / HTTP API | Storage backend hooks, post-operation callbacks, or polling |
| Native filesystem access | Filesystem watch subscription (inotify, fanotify, or fsnotify) |
For native filesystem access, the Runtime watches authorized directories and reacts to filesystem events. Implementations SHOULD debounce rapid successive writes to the same file before triggering replication.
In all cases, the Runtime MUST capture file creation, content update, and file deletion. Detected changes trigger the replication flow described in section 6.3.
6.5 Large File Handling
Files exceeding a configurable chunk size MUST be split into chunks for replication transfer. The default chunk size is 4 MB.
| Field | Description |
|---|---|
chunk_index | Zero-based index of the chunk |
chunk_count | Total number of chunks |
chunk_hash | SHA-256 hash of the individual chunk |
RPC transfers chunks sequentially or in parallel. The receiving node assembles chunks and verifies the final content_hash before marking the replica as complete.
6.6 Replication Scope
File replication is scoped to an information space. File objects belong to the information space configured for the application. Changes are propagated only to nodes participating in the same information space.
Files MUST NOT be replicated across information space boundaries.
When replication operates in user-level mode, the Runtime synchronizes only files the user is authorized to access during background replication. See Data Access Control for replication authorization contexts.
7. Consistency Model
7.1 Eventual Consistency
Distributed file storage provides eventual consistency by default. All replicas converge to the same file content given sufficient time and network connectivity.
7.2 Version Tracking
Every file object carries a version integer and content_hash managed by the Runtime. These are used for:
- detecting concurrent modifications
- verifying content integrity after transfer
- identifying when CRDT merge is required during synchronization
7.3 Integrity Verification
After receiving file content, the Runtime MUST verify the content_hash matches the SHA-256 hash of the received content. Mismatched content MUST be rejected and the transfer retried.
8. Conflict Resolution
Concurrent file updates are resolved with CRDT merge. File metadata fields merge structurally; file content converges through a Register CRDT — one current blob per file object, with deterministic tie-breaking from causal metadata (not wall-clock timestamp comparison alone).
8.1 Concurrent Update Detection
A concurrent update occurs when two nodes modify the same file object before replication completes, producing different metadata or content_hash values at the same logical object id.
The Replication engine detects concurrent modifications during synchronization using version and hash comparison via platform.replication.compare RPC calls.
8.2 CRDT Merge (Default)
| Strategy | Description |
|---|---|
| CRDT merge | Metadata fields merge per declared CRDT types; content converges via Register CRDT (default) |
| Manual | Automatic merge insufficient; flagged for operator review via platform.replication.resolve |
The default strategy is CRDT merge.
For metadata fields (mime_type, path attributes, custom tags), the Runtime applies the same CRDT type model as Distributed Database — CRDT merge. Binary content uses a Register CRDT — concurrent full-file replacements converge to one authorized value per merge rules.
8.3 Merge Flow
9. Offline Operation
9.1 Local Storage
Applications MAY store and read files locally while disconnected from remote nodes. All changes are persisted in the local storage backend and tracked by the Replication engine.
9.2 Synchronization on Reconnect
When connectivity is restored, the Runtime selects a synchronization strategy based on how long the node was offline.
| Offline duration | Strategy | Description |
|---|---|---|
| Short | Incremental | Replay pending changes and fetch missed notifications via platform.replication.fetch |
| Extended | Full state | Request complete file set via platform.replication.sync and platform.replication.snapshot |
Incremental Synchronization
Incremental synchronization is the default for short offline periods. The Runtime:
- sends messaging notifications for all pending local file changes
- fetches missed remote changes via
platform.replication.fetchRPC - detects concurrent updates and merges replicas via CRDT
- converges all replicas to a consistent state
Full State Synchronization
When a node has been offline for an extended period, incremental synchronization is insufficient. Change notifications sent while the node was unreachable are not retained — see Messaging section 8.3.
In this case, the node requests a full copy of all file objects from a peer instead of synchronizing individual change messages:
- the reconnecting node sends
platform.replication.syncvia Messaging - the peer transfers all file objects via
platform.replication.snapshotRPC - the reconnecting node applies the received state to its local storage backend
- the Runtime replays local pending offline changes and merges concurrent updates via CRDT
- incremental synchronization resumes for subsequent changes
A node SHOULD use full state synchronization when:
| Trigger | Description |
|---|---|
| Extended offline | Node was unreachable longer than a configurable threshold (recommended default: 24 hours) |
| First join | Node has no local replica for the information space |
| Large divergence | Local state is significantly behind peers |
| Operator request | Manual resync initiated by the node operator |
When replication operates in user-level mode, the snapshot MUST include only files the user is authorized to access. See Data Access Control.
Full state transfer MAY be split into multiple chunked RPC responses. The Runtime MUST verify file content_hash integrity after transfer completes.
9.3 Pending Changes
The Runtime MUST maintain a local log of file changes made while offline. This log is local to each node and tracks which file objects need synchronization.
9.4 Future: Flexible Full Synchronization
The baseline model in sections 9.2–9.3 transfers complete file objects (metadata and content) during extended offline recovery. For large files, media libraries, and storage-constrained user devices, future implementations SHOULD support flexible full synchronization without always copying every byte during initial sync.
| Mechanism | Description | Use case |
|---|---|---|
| Lazy content loading | Replicate file metadata locally; download content bytes from a source peer when the application reads the file or the Runtime prefetches | Attachments, images, video, large exports |
| Paginated snapshot | platform.replication.snapshot returns file objects in pages (cursor + limit); content MAY be omitted per page when using metadata-first mode | Reconnect after long offline period on mobile |
| Metadata-first replication | Replicate headers only — UUIDv7, path, content_hash, size, mime_type, version, and other metadata fields — with content_status: deferred until an explicit or lazy fetch | Directory listings and sync progress before bulk download |
A file header (metadata without local content) MUST still include enough information for access control evaluation, concurrent update detection (version, content_hash), and UI listing. Content fetch MUST verify content_hash after download.
On-demand content fetch
When metadata is local but content is deferred, the Runtime MUST obtain bytes through:
platform.replication.fetchfor a single file object, or- a content continuation RPC (future
platform.replication.fetch_contentor equivalent chunked transfer)
The source peer MUST authorize the fetch before sending content (see Data Access Control — Source-Side Authorization). Partial local files MUST NOT be served to applications as complete until content_hash verification succeeds.
Behaviour requirements (planned)
Future implementations MUST:
- support paginated full-state sync sessions alongside monolithic snapshot (sections 9.2–9.3)
- allow operators to cap concurrent content downloads and prefetch bandwidth
- keep files readable from cache when content is already local; trigger fetch only when metadata exists but content is deferred
- resume incremental synchronization after metadata-first or paginated full sync completes
Applications MAY request prefetch of deferred content (for example “download all attachments in this thread”). Prefetch MUST respect the same access policies and operator bandwidth limits.
This section describes planned protocol capability. The reference MVP MAY transfer small attachments inline; large-file lazy sync is expected in later platform hardening.
See Distributed Database — Flexible full synchronization for the structured-record equivalent.
10. Export, Backup, and Portability
File storage follows the same portability model as Distributed Database §10. Operators MUST be able to back up, export, and restore file objects without a vendor-controlled gate.
10.1 Operator Backup
Node backups MUST include, for each authorized information space:
| Component | Description |
|---|---|
| File metadata | UUIDv7 id, path, mime_type, version, content_hash, and custom metadata fields |
| File content | Bytes stored in the local storage backend (filesystem path, blob store key, or embedded blob) |
| Consistency | Backup SHOULD capture metadata and content in a consistent set — partial files MUST NOT be marked complete |
When the storage backend is a local filesystem, operators MAY use filesystem-level snapshots in addition to Runtime-coordinated export. The Runtime MUST still map paths to file object UUIDs on restore.
10.2 Portable Export
Portable export archives MUST:
- include file metadata and content (or content-addressed blobs referenced by
content_hash) - enforce source-side authorization — unauthorized files are omitted
- scope to a single
information_space_id - preserve UUIDv7 file object identifiers
Recommended packaging: a manifest plus a files/ directory of blobs named by UUIDv7 or content hash, bundled in tar or zip alongside the database export manifest from Distributed Database §10.2.
10.3 Restore and Migration
On node migration or import, the Runtime MUST verify content_hash after writing content bytes. Import MUST reject or quarantine objects whose hash does not match metadata.
Space fork (export from space A, import into new space B) follows Distributed Database §10.4. File UUIDs MAY be preserved within the imported archive; access policies and membership in space B require administrator setup.
The reference MVP MAY defer full file export/import; operator backup of attachment storage MAY be limited to filesystem snapshots when file replication is partial.
11. Information Space Scoping
File objects are scoped to an information space. The Runtime MUST verify application authorization for the information space before allowing file operations, as defined in the Node Discovery specification.
File replication propagates changes only within the authorized information space.
12. Access Control
Access to file objects is governed by the Data Access Control specification. Before reading or writing files, the Runtime verifies the user's cryptographic signature, approval status, and access policies for the target resource scope (path prefix, metadata field, or file condition).
Access control is enforced locally on each node. Replicated files are subject to the same access policies on every replica.
13. Security Requirements
Implementations MUST:
- assign UUIDv7 identifiers to all file objects at creation
- compute and verify SHA-256
content_hashfor all file content - enforce information space scoping on all file operations
- enforce access control before granting read or write access
- encrypt file content in transit during replication (via transport layer)
- sign all replication control messages (messaging and RPC)
Implementations MUST NOT:
- expose replication internals to applications
- replicate files across information space boundaries
- grant access to files without evaluating access control policies
- allow applications to manipulate
versionor replication metadata directly
14. Examples
14.1 Store a File
1. Application calls store({ path: "/documents/report.pdf", content: <bytes> })
2. Runtime assigns UUIDv7: 0195a3f0-9200-7899-cdef-000000000012
3. Runtime computes content_hash (SHA-256)
4. File written to local storage backend (version: 1)
5. Replication engine notifies peers via messaging
6. Remote nodes fetch metadata and content via RPC
7. Remote nodes verify content_hash and store locally
14.2 Local-First Read
1. Application calls read("0195a3f0-9200-7899-cdef-000000000012")
2. Runtime returns file content from local storage backend
3. No network call required
14.3 Large File Replication
1. Application stores 50 MB file
2. Runtime splits into 13 chunks (4 MB each)
3. Messaging notification sent to peers
4. Remote node fetches chunks via RPC (parallel transfer)
5. Remote node assembles chunks and verifies content_hash
6. File available locally on remote node
14.4 Offline Store and Sync
1. Node A is offline
2. Application stores file (version: 1)
3. Change recorded in local pending log
4. Node A reconnects
5. Runtime sends messaging notifications
6. Remote nodes fetch file via RPC
7. All replicas converge
14.5 Native Filesystem Write and Sync
1. Application writes /data/spaces/orders/documents/report.pdf directly
2. Runtime detects IN_MODIFY via inotify (or equivalent fsnotify event)
3. Runtime assigns UUIDv7 and computes content_hash
4. Replication engine notifies peers via messaging
5. Remote nodes fetch file content via RPC
6. Remote nodes write to their watched directory
7. Application on remote node reads file via native filesystem path