Node Connectivity
Maturity: Draft — not yet validated by a reference implementation. Subject to change during Runtime development.
See Protocol Specifications for reading order and the full specification index.
Abstract
This specification defines the Faberio Node Connectivity service — the mechanism by which nodes establish network connections for communication within an information space.
Node connectivity supports direct connections, NAT traversal, and peer-to-peer (P2P) connectivity for nodes without dedicated public IP addresses. Operators, information space administrators, and (where permitted) users MAY configure STUN and TURN infrastructure. The Runtime defines limits and degraded connectivity behaviour when configured or public relays are unavailable. The model is inspired by WebRTC ICE negotiation, adapted for the Faberio decentralized runtime environment.
1. Scope
This specification defines:
- connection establishment between nodes
- transport abstraction
- connectivity candidates and negotiation
- NAT traversal and hole punching
- relay fallback for restrictive networks
- STUN/TURN server configuration (operator, federated information-space, and user device)
- relay usage limits and degraded connectivity when relays are unavailable
- connection lifecycle and recovery
- security requirements for transport connections
This specification does not define:
- node discovery (see Node Discovery)
- node identity and authentication (see Identity Model)
- synchronous request-response communication (see Messaging and RPC)
- specific codec or media handling
- STUN/TURN server implementation
2. Motivation
Nodes in the Faberio network often operate on consumer hardware, home networks, mobile devices, and edge infrastructure. Many of these environments place nodes behind NAT without a dedicated public IP address.
Traditional client-server models assume at least one side has a reachable public endpoint. Faberio requires symmetric peer-to-peer connectivity where both nodes may be behind NAT.
Node connectivity solves this by:
- gathering multiple connectivity candidates per node
- negotiating a working path between peers
- performing NAT hole punching for direct P2P connections
- falling back to relay servers when direct connectivity is impossible
This enables nodes to communicate regardless of network topology, matching the platform's goal of operating across heterogeneous infrastructure.
3. Concepts
3.1 Transport Connection
A transport connection is an established network channel between two nodes over which the Faberio Protocol operates. All platform services — messaging, RPC, replication — use transport connections.
3.2 Connectivity Candidate
A connectivity candidate is a potential network path through which two nodes may communicate. Candidates are collected during connection establishment and tested to find a working route.
3.3 Signaling
Signaling is the exchange of connectivity metadata between nodes during connection establishment. Signaling messages are transmitted over an existing authenticated Faberio connection or through a bootstrap peer that both nodes can reach.
3.4 Connection Negotiation
Connection negotiation is the process by which two nodes exchange candidates, test connectivity, select a working path, and establish an encrypted transport channel.
3.5 Connectivity Infrastructure Policy
A connectivity infrastructure policy defines which STUN and TURN servers a node MAY use, whether public relay services are permitted, and limits on relay usage. Policies MAY be set at three levels — node operator, information space (federated), and user device — as defined in section 7.6.
3.6 Relay Degraded Mode
Relay degraded mode is the Runtime state in which relay candidates cannot be allocated (because of policy, quota limits, or server unavailability) but direct connectivity paths MAY still be attempted. Application data continues to operate locally; cross-node operations that require a failed transport connection queue or retry according to section 7.7.
4. Connectivity Architecture
5. Connectivity Candidates
5.1 Candidate Types
Implementations MUST support the following candidate types:
| Type | Description | Example |
|---|---|---|
host | Local network interface address | 192.168.1.10:7700 |
server-reflexive | Public address discovered via STUN | 203.0.113.42:55123 |
relay | Address allocated on a relay server | 198.51.100.5:60001 |
The host type represents direct local addresses. The server-reflexive type represents the node's public-facing address as seen through NAT, discovered using a STUN-like mechanism. The relay type represents a relayed address on a TURN-like server used when direct connectivity fails.
5.2 Candidate Format
Each connectivity candidate MUST contain:
| Field | Type | Description |
|---|---|---|
type | string | host, server-reflexive, or relay |
transport | string | Transport protocol (tcp, quic, udp) |
address | string | IP address or hostname |
port | integer | Port number |
priority | integer | Candidate priority for path selection |
5.3 Candidate Gathering
When initiating or accepting a connection, a node MUST gather candidates in the following order:
- Host candidates — addresses of all local network interfaces
- Server-reflexive candidates — public address discovered via configured STUN servers
- Relay candidates — addresses allocated via configured TURN servers (if direct paths are unavailable)
Nodes behind NAT without a public IP address typically have only host (private) and server-reflexive candidates. Connection establishment relies on NAT hole punching between these candidates.
6. Connection Establishment
6.1 Overview
Connection establishment follows an offer/answer model with candidate exchange, similar to WebRTC ICE:
6.2 Connectivity Offer
A node initiates connection by sending a connectivity offer containing:
| Field | Description |
|---|---|
initiator_node_id | Node ID of the initiating node |
target_node_id | Node ID of the target node |
information_space_id | UUIDv7 of the information space |
candidates | Array of connectivity candidates |
signature | Ed25519 signature over the offer payload |
6.3 Connectivity Answer
The target node responds with a connectivity answer containing:
| Field | Description |
|---|---|
responder_node_id | Node ID of the responding node |
information_space_id | UUIDv7 of the information space |
candidates | Array of connectivity candidates |
signature | Ed25519 signature over the answer payload |
6.4 Connectivity Checks
After candidate exchange, both nodes perform connectivity checks — test messages sent between candidate pairs to determine which path works.
Checks MUST be performed for all candidate pairs, starting with the highest priority. The first successful pair is selected for the transport connection.
6.5 Authentication
Once a candidate pair is selected, nodes MUST perform mutual authentication as defined in the Identity Model before any application data is transmitted.
The transport connection MUST be encrypted. Implementations MUST use TLS 1.3 or QUIC with TLS 1.3 for all transport connections.
7. NAT Traversal
7.1 Problem
Nodes behind NAT have private IP addresses that are not directly reachable from the Internet. Both nodes may be behind different NAT devices, making direct connection appear impossible.
Node A (192.168.1.10) → NAT A (203.0.113.10) → Internet ← NAT B (203.0.113.20) ← Node B (192.168.2.20)
7.2 STUN — Server-Reflexive Candidates
Implementations SHOULD use STUN-like servers to discover server-reflexive candidates. A STUN request reveals the node's public IP and port as mapped by its NAT device.
STUN servers MAY be operated by the node operator, the information space administrator, or a public STUN service. STUN servers do not relay data — they only assist in address discovery.
7.3 NAT Hole Punching
When both nodes have server-reflexive candidates, implementations MUST attempt NAT hole punching — simultaneous outbound connections from both nodes to create a direct P2P path through both NAT devices.
Hole punching works for most cone and restricted cone NAT types. Symmetric NAT may require relay fallback.
7.4 TURN — Relay Fallback
When direct connectivity and hole punching fail, implementations MUST fall back to a relay candidate via a TURN-like server.
Relay connections route all traffic through the relay server. This guarantees connectivity at the cost of additional latency and bandwidth through the relay.
Relay servers MAY be operated by node operators or information space administrators. Relay usage SHOULD be configurable — operators may prefer direct P2P and only enable relay when necessary.
7.5 Path Selection Priority
Implementations MUST select connectivity paths in the following priority order:
- Direct host-to-host — both nodes on the same local network
- Server-reflexive P2P — direct connection via NAT hole punching
- Relay — traffic routed through a TURN-like server from the effective server list (section 7.6)
7.6 STUN/TURN Server Configuration
Faberio does not mandate a single global STUN/TURN provider. Each deployment MAY use operator-owned, information-space federated, user-configured, or public servers, subject to policy and limits defined in this section.
7.6.1 Configuration levels
| Level | Set by | Scope | Replication |
|---|---|---|---|
| Node operator | Operator (NixOS module, container env, local Runtime config) | All connectivity on that physical node | Local only |
| Information space | Space administrator (via replicated configuration record) | All approved nodes and user devices in the space | Node-level background replication |
| User device | End user on a device node | That user's device node only | Local only; MUST NOT override space or operator deny rules |
The effective connectivity infrastructure policy for a node in an information space is computed by merging the three levels (section 7.6.4).
7.6.2 Federated information-space configuration
Information space administrators MAY publish a connectivity infrastructure record stored as replicated data within the information space (same trust model as node membership and access policies).
The record SHOULD include:
| Field | Type | Description |
|---|---|---|
information_space_id | UUIDv7 | Target information space |
stun_servers | array | Ordered STUN server URIs (stun:host:port or stuns:host:port) |
turn_servers | array | TURN server entries (see section 7.6.3) |
allow_public_stun | boolean | Whether built-in or operator-default public STUN MAY be used when not listed |
allow_public_turn | boolean | Whether public TURN fallback MAY be used when configured relays fail |
allow_user_relay_config | boolean | Whether user devices MAY append personal STUN/TURN entries |
relay_limits | object | Space-wide relay limits (section 7.6.5); MAY be capped by operator limits |
updated_at | timestamp | Last policy change |
signature | bytes | Ed25519 signature by an authorized administrator |
Only users with write access to the connectivity infrastructure record (as defined by the application and Data Access Control) MAY change federated settings. Changes propagate to all nodes through node-level background replication.
Federated configuration enables consistent relay policy across a community or organisation without relying on third-party public relays.
7.6.3 Server and credential format
Each TURN server entry MUST support:
| Field | Type | Description |
|---|---|---|
urls | array | One or more TURN URIs (turn:, turns:) |
username | string | Optional long-term credential username |
credential | string | Optional password or time-limited secret |
credential_type | string | password or oauth (if supported by implementation) |
Credentials MUST NOT be stored in plaintext in replicated database records visible to unapproved users. Implementations SHOULD:
- store federated TURN secrets in operator-provisioned local secret stores referenced by record ID, or
- use time-limited TURN credentials minted by the operator's credential service, or
- encrypt credential fields for replication using keys available only to approved nodes in the information space
STUN entries are URIs only; TURN entries include authentication when required by the relay operator.
Node operator configuration uses the same schema and MAY be supplied through deployment tooling (for example NixOS module options or environment variables).
User device configuration MAY supply additional stun_servers and turn_servers entries in local device settings when allow_user_relay_config is true in the effective policy.
7.6.4 Policy merge and precedence
When gathering candidates, the Runtime MUST compute the effective server list and flags using the following rules:
- Deny wins — if the node operator sets
allow_public_turn: false, neither federated nor user settings MAY enable public TURN fallback. - Operator floor — operator
relay_limitsMUST NOT be exceeded by federated or user limits (operator MAY set stricter caps). - Server list merge — STUN/TURN URIs are merged in order: information space defaults, then node operator additions, then user device additions (if permitted). Duplicate URIs MUST be deduplicated.
- Operator override — the operator MAY set
replace_space_servers: trueto use only operator-configured servers (ignoring federated lists except for signaling/bootstrap peers). - Public fallback — public STUN/TURN MAY be appended only when both the effective
allow_public_stun/allow_public_turnflags are true and no explicit policy forbids them.
Implementations MUST document which optional public STUN/TURN endpoints the reference Runtime uses when allow_public_* is true, so operators can disable them explicitly.
7.6.5 Relay limits
Implementations MUST support configurable limits at node operator level. Information space administrators MAY publish additional limits in the federated record; user devices inherit the effective limits.
| Limit | Description | Default behaviour when exceeded |
|---|---|---|
max_concurrent_relay_sessions | Maximum active TURN allocations per node | Deny new relay allocations; continue direct path attempts |
max_relay_bandwidth_bps | Aggregate relay throughput cap for the node | Throttle or refuse new relay sessions; existing sessions MAY be closed gracefully |
max_relay_bytes_per_day | Daily relay traffic budget | Enter relay degraded mode until next budget window |
max_stun_requests_per_minute | STUN discovery rate limit | Back off STUN gathering; use cached server-reflexive mappings when valid |
max_relay_session_duration | Maximum lifetime of a single TURN allocation | Refresh if allowed; otherwise close and renegotiate |
When a limit is exceeded, the Runtime MUST:
- emit an operator-visible connectivity warning (log, metrics, or admin API event)
- NOT allocate new relay candidates until quota allows
- prefer direct and server-reflexive paths before retrying relay
Limits apply per physical node unless the deployment explicitly tracks per-user device quotas in application configuration.
7.6.6 Public relay availability
Public STUN and TURN services are optional and MUST NOT be assumed available. Implementations MUST treat public endpoints as last-resort fallback when enabled by policy — never as the only connectivity path.
If public STUN is disabled or unreachable:
- gathering of server-reflexive candidates MAY fail or use stale mappings
- host candidates and signaling through bootstrap peers MUST still be attempted
- NAT hole punching MAY still succeed when at least one side has a reachable reflexive address from a configured STUN server
If public TURN is disabled or all configured TURN servers are unreachable:
- the Runtime MUST NOT silently substitute a different public relay without
allow_public_turn: true - relay candidate gathering MUST fail explicitly
- connectivity negotiation continues with host and server-reflexive paths only (section 7.8)
7.7 Behaviour When Relays Are Unavailable
When no working TURN server is available — because of policy, quota, network failure, or symmetric NAT blocking direct paths — the Runtime enters relay degraded mode for affected peer pairs.
7.7.1 Connection establishment
For a pair of nodes where relay is required but unavailable:
- The Runtime MUST complete connectivity checks for all non-relay candidate pairs.
- If no pair succeeds, the transport connection MUST fail with a connectivity error indicating
relay_unavailable,relay_quota_exceeded, ordirect_path_failedas appropriate. - Implementations MUST NOT block local application reads and writes; only cross-node transport over that peer pair is affected.
The connectivity error SHOULD be observable to applications through Runtime status APIs (exact surface is implementation-defined).
7.7.2 Retry and recovery
Implementations MUST retry failed connections using:
- last known working candidate pair (if any)
- full re-gathering with configured STUN servers
- relay re-allocation when servers become reachable or quota resets
Retry intervals SHOULD use exponential backoff with a configurable maximum. Default initial retry interval SHOULD be 5 seconds; maximum SHOULD be 5 minutes unless the operator configures otherwise.
7.7.3 Impact on platform services
| Service | Behaviour in relay degraded mode |
|---|---|
| Background replication | Queues outbound changes for unreachable peers; continues for peers with working transport |
| Messaging / RPC | Delivery to unreachable peers fails or queues until connectivity restores |
| Signaling | MUST continue via any reachable bootstrap or mutual peer |
| Local-first reads/writes | Unaffected on the local node |
If at least one always-on server node with a public or relay-reachable address participates in the information space, user devices behind symmetric NAT MAY still synchronize indirectly through that node (node-level replication), even when direct device-to-device transport fails.
7.7.4 Operator and user expectations
Operators SHOULD deploy at least one TURN server under their control for information spaces that include user devices behind restrictive NAT, rather than relying on public relays.
User devices MUST NOT assume relay connectivity. Applications SHOULD surface connectivity state when messaging or sync is delayed because of relay_unavailable.
8. Signaling
8.1 Signaling Channel
Connectivity offers and answers are exchanged through a signaling channel. The signaling channel is an existing authenticated Faberio connection to a mutually reachable peer.
Signaling paths include:
- a direct connection already established between the two nodes
- a bootstrap peer configured for the information space
- a third peer reachable by both nodes that forwards signaling messages
8.2 Signaling Without Public Endpoints
Nodes without any public endpoint rely entirely on signaling through reachable peers and P2P candidate exchange. A node behind NAT does not need a public IP in its discovery record — connectivity is established through candidate negotiation.
Discovery records MAY include partial or private endpoint information. The connectivity engine supplements discovery endpoints with gathered candidates during connection establishment.
9. Connection Lifecycle
9.1 Establishment
A transport connection is established when:
- candidate negotiation completes
- a working candidate pair is selected
- mutual authentication succeeds
- the encrypted channel is active
9.2 Maintenance
Implementations SHOULD send keepalive messages on idle connections to prevent NAT mapping expiration. The default keepalive interval SHOULD be 30 seconds.
9.3 Reconnection
When a connection is lost, the Runtime MUST attempt reconnection using the following strategy:
- retry the last known working candidate pair
- re-gather candidates and perform full negotiation
- fall back to relay if direct paths fail
9.4 Termination
A node MAY gracefully close a connection by sending a signed disconnect message. The peer MUST release associated resources upon receiving a disconnect message.
10. Information Space Scoping
Transport connections are scoped to an information space.
Connectivity offers and answers MUST include information_space_id. Connections established for one information space MUST NOT be reused for another.
A node that participates in multiple information spaces maintains separate transport connections per space.
11. Security Requirements
Implementations MUST:
- verify Ed25519 signatures on connectivity offers and answers
- perform mutual authentication after candidate pair selection
- encrypt all transport connections using TLS 1.3 or QUIC
- scope connections to information spaces
- reject connectivity messages with mismatched
information_space_id - honour effective connectivity infrastructure policy (section 7.6) — MUST NOT use public relays when
allow_public_turnorallow_public_stunis false - protect TURN credentials at rest and in transit; MUST NOT replicate plaintext relay secrets to unapproved nodes
Implementations MUST NOT:
- transmit application data over unauthenticated connections
- transmit application data over unencrypted channels
- accept connectivity offers from nodes outside the information space
- reuse relay servers to intercept or inspect application data
- bypass operator or information-space deny rules via user device configuration
12. Examples
12.1 Direct Connection — Same LAN
Node A: 192.168.1.10 (host candidate)
Node B: 192.168.1.20 (host candidate)
1. Node A sends connectivity offer with host candidate
2. Node B responds with host candidate
3. Connectivity check succeeds on local network
4. Direct connection established (no NAT traversal needed)
12.2 P2P Connection — Both Behind NAT
Node A: 192.168.1.10 → NAT → 203.0.113.10:55123 (server-reflexive)
Node B: 192.168.2.20 → NAT → 203.0.113.20:44156 (server-reflexive)
1. Both nodes gather host and server-reflexive candidates via STUN
2. Candidates exchanged via signaling through bootstrap peer
3. Both nodes send simultaneous outbound packets (hole punching)
4. Direct P2P connection established through both NATs
12.3 Relay Fallback — Symmetric NAT
Node A: server-reflexive candidate (hole punching fails)
Node B: server-reflexive candidate (hole punching fails)
1. Candidate exchange and connectivity checks fail for direct paths
2. Both nodes allocate relay candidates via TURN server
3. Connectivity check succeeds via relay
4. Connection established through relay server
12.4 Operator TURN Only — Public Relays Disabled
Information space policy:
allow_public_stun: false
allow_public_turn: false
turn_servers: [ turn:relay.example.org:3478 (operator credentials) ]
Node A and Node B: both behind symmetric NAT; direct hole punching fails
1. Both nodes gather relay candidates only from relay.example.org
2. relay.example.org is unreachable (network outage)
3. Connectivity checks fail for relay paths
4. Runtime reports connectivity error relay_unavailable
5. Local reads/writes continue; replication to the peer queues
6. Operator restores relay or adds reachable TURN to federated config
7. Runtime retries with exponential backoff; relay path succeeds after recovery
12.5 Federated Defaults with User Device Extension
Information space policy:
stun_servers: [ stun:stun.space.example:3478 ]
allow_user_relay_config: true
User device adds personal TURN for mobile network:
turn_servers: [ turn:turn.user.example:3478 ]
1. Effective STUN list = space STUN + user STUN (deduplicated)
2. Effective TURN list = space TURN (if any) + user TURN
3. Direct P2P attempted first; user TURN used only if direct fails and within relay limits