backspace/docs/systems/federation.md

Federation System (Server-to-Server)

Companion spec: This document covers S2S (server-to-server) federation — the relay protocol, HMAC auth, identity resolution, and background workers. For the client-side multi-instance architecture (how the web/desktop app connects to multiple instances, federated account creation, origin-aware routing), see client-federation.md. Both systems work together.

Source files:

• packages/server/src/routes/federation.ts -- API endpoints (peer handshake, relay, sync) + all inbound event processors + identity resolution functions
• packages/server/src/utils/federationAuth.ts -- HMAC signing, verification, header parsing, getOurOrigin()
• packages/server/src/utils/federationOutbox.ts -- Event queuing, coalescing, relay payload construction, mutation log, participant/target resolution
• packages/server/src/utils/federationLookup.ts -- HMAC-signed remote-user lookups: lookupRemoteUser (by username) and lookupRemoteUserByHomeId (reverse lookup, used by stub backfill)
• packages/server/src/utils/federationPresence.ts -- S2S presence relay: queuePresenceRelay, snapshotPresenceForPeer (relationship-scoped), markPeerStubsOffline
• packages/server/src/utils/federationStubBackfill.ts -- Heals legacy snowflake-named replicated-user stubs by reverse-looking-up the canonical username via the peer
• packages/server/src/utils/federationWorker.ts -- Background workers: outbox delivery, file download, health check, janitor, initial sync
• packages/server/src/utils/storageJanitor.ts -- Federation GC: outbox expiry, mutation log retention, file queue cleanup, DM channel purge
• packages/server/src/routes/social.ts -- Friend request/accept/cancel/remove endpoints that queue federation events
• packages/server/src/routes/dm.ts -- DM REST endpoints that queue federation events (message relay, group lifecycle)
• packages/server/src/ws/events.ts -- WebSocket event handlers that queue DM message/reaction relay events
• packages/web/src/utils/profileSync.ts -- Client-side profile sync via LWW timestamps (not S2S relay)
• packages/web/src/utils/identity.ts -- Client-side federated identity resolution helpers

DB tables: federation_peers, federation_outbox, federation_file_queue, federation_mutation_log, plus users (identity), dm_channels/dm_members/dm_messages (DM federation), friends/friend_requests (friend federation), attachments (file replication). See docs/systems/database.md for full schemas.

---

Architecture Overview

Backspace federation is peer-to-peer with no central authority. Each instance maintains its own copy of all data. Peers exchange real-time events for DMs and friendships via a signed relay protocol.

Canonical identity: The (homeUserId, homeInstance) pair is globally unique. Local users have homeInstance = NULL and homeUserId = NULL. Federated users are represented as replicated user stubs -- minimal user records with passwordHash = '!federation-replicated' (bcrypt never produces this value, so login is impossible).

Trust model: Symmetric shared-secret HMAC. Both peers share the same 256-bit secret. Events are attributed to users by homeUserId + homeInstance in the payload, with authority checks verifying the source instance matches the claimed origin of the acting user.

---

1. Peer Handshake & Discovery

2-Phase Flow

Phase 1 -- Initiate (POST /api/federation/peer/initiate)

• Auth: JWT + admin role required
• Validates remoteOrigin is a well-formed HTTP(S) URL via validateOrigin()
• Prevents self-peering (localOrigin === remoteOrigin)
• Handles existing peers: active -> return 200, pending -> return 409, revoked -> delete and re-initiate
• Generates HMAC secret: generateHmacSecret() -> randomBytes(32).toString('hex') (256-bit)
• Generates challenge: randomBytes(16).toString('hex') (128-bit, currently unused by acceptor)
• Creates local peer record with status='pending'
• POSTs to {remoteOrigin}/api/federation/peer/accept with { sourceOrigin, challenge, hmacSecret }
• Timeout: 10 seconds (AbortSignal.timeout)
• On remote 200 (accepted): verifies before activating — performs a signed fetchPeerEpoch (POST /api/federation/epoch) round-trip with the just-negotiated secret to PROVE the responder actually adopted it (a responder that returns 200 without adopting the secret is otherwise indistinguishable from a healthy fresh peering — this is the split-brain the honest contract prevents). If fetchPeerEpoch succeeds, updates local peer to status='active', stores the cryptographically-verified epoch as peer_instance_id (authoritative over the unverified handshake-response body), sets lastSeenAt, broadcasts federation_peers_changed, and returns 200 with { peer, verified: true }. If it returns null (secret doesn't authenticate / /epoch absent / unreachable), parks the peer in status='needs_attention' with needs_attention_reason='repeer_incomplete' and returns 200 with { peer, verified: false } — never a silent false-active. See "Trust re-establishment contract" below.
• On remote 409 PEER_EXISTS_RESET_REQUIRED (responder already holds peering for us and honestly refuses to rekey): deletes its own pending row — keeping the local slot clean so the remote's own later Re-peer can land on a fresh responder slot — and returns 409 { code: 'PEER_EXISTS_RESET_REQUIRED', error }. See "Trust re-establishment contract" below.
• On remote 202 (queued for remote admin approval): transitions local peer to status='awaiting_approval' (does not activate), broadcasts federation_peers_changed, returns 202 with { peer }. Without this branch response.ok would be true and the local peer would flip to active while the remote had us pending — a transient local-active / remote-pending split that only self-healed when the remote admin approved. Mirrors the auto-peer 202 branch in federationPeering.ts:performHandshake.
• On remote 403 / other non-2xx: deletes pending peer, returns 502 with the remote's error message
• On network error / timeout: deletes pending peer, returns 502 (network) or 504 (timeout)

Phase 2 -- Accept (POST /api/federation/peer/accept)

• Auth: none (first contact -- no JWT, no HMAC)
• Rate-limited: 10 requests per minute per IP (in-memory sliding window, buckets cleaned every 60s)
• Validates sourceOrigin, challenge, hmacSecret, and (optional) instanceName from body
• Handles existing peers: active or needs_attention -> return 409 PEER_EXISTS_RESET_REQUIRED (honest refusal — the anti-hijack guard still does NOT adopt the caller's hmac_secret; only the reported status/body changed from the old false 200 {accepted:true}), revoked -> return 403, pending -> update with new secret and activate
• Epoch-mismatch reset detection (markPeerReset) still fires on the active/needs_attention path before the 409 is returned — detection is layered on top of the unchanged anti-hijack guard
• New peer: creates record with provided hmacSecret, sets status='active'
• Returns { accepted: true, instanceName: <ourName | null>, instanceId: <ourEpoch> } on true activation; on the existing-active/needs_attention refusal returns 409 { accepted: false, code: 'PEER_EXISTS_RESET_REQUIRED', error, instanceName, instanceId } — see "Instance name & epoch exchange" and "Trust re-establishment contract" below

Instance name & epoch exchange

The handshake is bidirectional for two pieces of metadata: the instance_name label rendered in the federation panel and in DM-call toasts (peerLabel), and the instance epoch (instance_id, this instance's persistent incarnation UUID minted by ensureDefaults, accessed via getInstanceId()). The epoch is the authenticated baseline used by the instance-epoch self-healing feature to detect a wipe-and-reinstall on the same domain (design: docs/superpowers/specs/2026-07-01-federation-instance-epoch-self-healing-design.md).

• Initiator → responder: the request body to /peer/accept carries { sourceOrigin, hmacSecret, instanceName, instanceId }. The responder reads instanceName → federation_peers.instance_name and instanceId → federation_peers.peer_instance_id on every state-mutating activation path: pending → active, awaiting_approval → active (token-valid and autoAccept-fallback), rejected → active (override), and new-peer create. The existing-peer refusal path for already-active and needs_attention peers (now an honest 409 PEER_EXISTS_RESET_REQUIRED, see "Trust re-establishment contract") does NOT overwrite — same security posture that already refuses to overwrite hmac_secret on these paths from an unauthenticated request. (The idempotent guard's detection of a changed epoch on that path is a later part of the self-healing feature; the handshake itself only writes the epoch on true activation.)

• Responder → initiator: the /peer/accept response body is { accepted: true, instanceName: <ourName | null>, instanceId: <ourEpoch> }. The initiator (performHandshake in utils/federationPeering.ts, /peer/initiate, and both /approval-requests/:id/approve handlers in routes/federation.ts) parses instanceName and instanceId and persists them alongside the status='active' write (peer_instance_id). Older peers that omit either field are tolerated — the respective column stays null (backstopped later by the deterministic epoch-refresh and relay-envelope population). Non-JSON bodies are tolerated defensively.

All four outbound /peer/accept senders (performHandshake, /peer/initiate, and the inbound + outbound /approve handlers) include instanceId: getInstanceId() in the request body, so a peer learns our epoch regardless of which path activated the relationship.

instance_name is cosmetic metadata, eventually-consistent. Anywhere peerLabel is rendered falls back to origin hostname when instance_name IS NULL. Instance renames do not currently re-broadcast — that's a separate, unimplemented feature. peer_instance_id is trust-consequential (only ever written from authenticated channels) — see the self-healing design spec for detection/heal semantics.

Secret Storage & Rotation

Both instances store the same HMAC secret. The initiating instance generates it and sends it in the accept request.

Rotation protocol: Either peer (or automatically on a configurable interval, default 90 days) can trigger rotation:

1. Initiator generates a new secret, stores it as pendingHmacSecret, and POSTs to {peer}/api/federation/peer/rotate signed with the current secret
2. Acceptor verifies the HMAC, stores pendingHmacSecret, and returns { accepted: true }
3. Both sides enter a 15-minute grace period where either secret is accepted for verification, while outbound requests are signed with the new secret
4. After the grace period, the health check worker promotes pendingHmacSecret → hmacSecret and clears the pending fields

Conflict guard: If pendingHmacSecret is already set, the rotate endpoint returns 409.

Schema columns: pending_hmac_secret (TEXT NULL), secret_rotation_at (INTEGER NULL), secret_rotated_at (INTEGER NULL), auto_rotate_interval_days (INTEGER NOT NULL DEFAULT 90).

Peer Status Lifecycle

                     ensurePeered
  (none) ──────────► pending ──────────► active ──────► needs_attention
      ▲                │                   │ ▲              │
      │                │  remote 202       │ │              │ admin Reset
      │                ▼                   │ │              ▼
      │          awaiting_approval         │ │           (deleted)
      │                │                   │ │
      │    ┌───────────┼──────────┐        │ │ N consecutive
      │    │           │          │        │ │ auth failures (401/403)
      │  accept     denied     expired     │ │
      │  (fresh)    (admin)   (janitor)    │ │ delivery failures
      │    │           │          │        ▼ │
      │    ▼           ▼          ▼     unreachable
      │  active    rejected   rejected     │
      │                                    │ health check OK
      │  auto-peer rejected                ▼
      │  (403 PEERING_REQUIRES_APPROVAL) active
      └──────────────────────────── rejected
                                      │ admin revoke (active)
                                      ▼
                                   revoked

Status	Outbox delivery	Health check	Relay accepts	Re-initiation	Admin clear
active	Yes	No	Yes	No (returns existing)	N/A
pending	No	No	No	No (returns 409)	N/A
awaiting_approval	No	No	No	Returns pending; no re-handshake	Yes (admin deletes)
unreachable	No (entries wait)	Yes — demand-driven probe (backoff while mail queued; 15-min backstop when silent)	Yes (resets to active)	No	N/A
needs_attention	No (entries bounded by TTL)	No	Yes (200 no-update, same as active)	No (admin must Reset first)	Yes (admin Reset deletes record)
revoked	No (entries purged)	No	No (returns 403)	Yes (old record deleted)	N/A
rejected	No	No	No	Yes (admin deletes record, then re-initiates)	N/A

PEER_UNREACHABLE_THRESHOLD

Defined in federationWorker.ts as 10. After 10 consecutive delivery failures for a peer, handleOutboxDeliveryFailure sets status = 'unreachable' and, on entry, resets the recovery pacing fields (probe_attempts = 0, last_probe_at = NULL) so the first probe fires immediately.

Demand-driven recovery (processRecoveryTick) — A dedicated recovery loop runs on a 5-second tick (RECOVERY_TICK_INTERVAL_MS). On each tick it scans unreachable peers and, for each, decides whether a probe is due:

• If the peer has ≥1 queued outbox row, the probe interval is RECOVERY_BACKOFF_MS = [30s, 1m, 5m, 15m] indexed (clamped) by probe_attempts — fast retries while mail is actually waiting.
• If the peer has no queued mail, it falls back to the HEALTH_CHECK_INTERVAL_MS (15-minute) backstop — there is nothing to deliver, so there is no urgency.
• A probe is due when last_probe_at IS NULL (immediate first probe on the unreachable transition) or now - last_probe_at >= interval.

When due, it calls probePeerReachable(origin) (utils/federationRecovery.ts) — an unauthenticated GET /api/instance/info with a 10-second timeout (no HMAC; reachability is not trust). On success it calls markPeerRecovered(peerId), which reverts the peer to active, zeroes consecutive_failures, refreshes last_seen_at, resets probe_attempts/last_probe_at, and broadcasts federation_peers_changed (via onPeerActivated). On failure it increments probe_attempts and stamps last_probe_at = now to advance the backoff.

processHealthCheckTick (15-minute interval, matching ROTATION_GRACE_PERIOD_MS) no longer owns recovery — it now handles secret-rotation grace-period finalization and auto-rotation only.

Auto-Peering

When the server needs to relay events to an instance it has not yet peered with, ensurePeered() in federationPeering.ts automatically initiates the handshake. This removes the requirement for an admin to manually initiate every peering relationship.

Integration points:

• Outbox worker (federationWorker.ts) — after delivering active peers, calls ensurePeered() for any pending-placeholder entries whose peer has not yet been activated.
• Connection flow (POST /api/federation/peer/ensure) — called by the client when establishing a cross-instance connection, ensuring the two instances are peered before any relay traffic is sent.

rejected status — Added to the peer lifecycle. Set when a remote instance explicitly rejects auto-peering with 403 PEERING_REQUIRES_APPROVAL. This status is sticky: no automatic retry occurs. An admin can clear it by deleting the peer record (then re-initiating), or by manually initiating via peer/initiate. An incoming peer/accept from a remote admin can also override rejected → active (treated the same as a pending record).

autoAcceptPeering instance setting — Controls whether POST /api/federation/peer/accept accepts unsolicited peering requests. Default: true. When false, the endpoint returns 403 PEERING_REQUIRES_APPROVAL for requests where no local pending record exists (i.e., a request the local admin did not initiate). The determination is made by checking the local peer table — not a client-provided flag.

Peer Approval Queue

When autoAcceptPeering is false and an instance calls POST /api/federation/peer/accept without a matching local pending record, the endpoint returns 202 Accepted and creates a row in peer_approval_requests instead of immediately peering. The requesting instance receives 202 (not an error), so it enters awaiting_approval status rather than rejected.

peer_approval_requests table — Holds incoming peering requests pending admin review:

• id — Snowflake PK
• origin — Requesting instance's origin URL (UNIQUE; only one pending request per origin)
• instance_name — Instance name sent by requester
• hmac_secret — Requester's HMAC secret; used to sign the denial notification
• requested_at / expires_at — Epoch ms; expiry is requested_at + 30 days
• approval_token — Single-use 64-hex-char random token issued in the 202 response and forwarded by /approve. See Approval Token Verification.

Approval flow — Admin approves via POST /api/federation/approval-requests/:id/approve. The handler dispatches on peer_approval_requests.direction:

Inbound (remote asked to peer with us):

1. A fresh federationPeer record is created (or existing rejected/awaiting_approval record is upserted) with status pending
2. A standard peer/accept handshake is sent to the requesting origin (forwarding the stored approvalToken per Approval Token Verification)
3. On 200 the local peer becomes active; on 202 the peer transitions to awaiting_approval (mutual-gate); the peer_approval_requests row is deleted in both cases

Outbound (local users asked us to peer with a remote — created by the gate in ensurePeered):

1. A fresh federationPeer row is created with status pending (HMAC generated at this moment — outbound queue rows store hmac_secret = NULL)
2. peer/accept is sent to the remote with no approvalToken (we are the initiator with no prior token from this remote)
3. On 200 the peer is promoted to active. onPeerActivated then fires fanoutOutboundSubscribers (see Outbound peering gate) which writes kind='approved' notifications for each subscriber and cascade-deletes the parent peer_approval_requests row. The handler does NOT duplicate this cleanup.
4. On 202 the peer transitions to awaiting_approval, captures the returned approvalToken, and the queue row + subscribers are LEFT INTACT — they wait for the eventual remote-admin approval. onPeerActivated is NOT called on this path.
5. On 4xx/5xx/network error the peer row is deleted; the queue row is left intact so the admin can retry. Response status is 502/503/504 accordingly.
6. Response body shape: { success, peerStatus: 'active' | 'awaiting_approval', peer? }. The peerStatus field is the outbound-only signal.

Denial flow — Admin denies via POST /api/federation/approval-requests/:id/deny. The handler dispatches on direction:

Inbound:

1. Server sends POST {origin}/api/federation/peer/denied signed with the requester's hmac_secret (from the approval request row)
2. Receiving instance transitions its local peer record from awaiting_approval → rejected
3. A local federationPeer record is upserted with status rejected to block future unsolicited requests from the same origin
4. The peer_approval_requests row is deleted

Outbound:

1. For each row in peer_approval_subscribers, a kind='denied' notification is inserted into peer_approval_notifications and a peering_notification_received WS event is sent to the user
2. The parent peer_approval_requests row is deleted (cascade clears subscribers)
3. No remote network call — the remote never knew we were considering this peer
4. federation_peers_changed is broadcast to admins so the queue UI refreshes

Expiry — The janitor (federationJanitor.ts) runs on its scheduled interval and deletes rows where expires_at < now. Expired requests do NOT create a rejected peer — the requesting instance can re-submit. Admin denial, by contrast, does create a rejected peer record, blocking re-requests until an admin clears it.

Pre-handshake guard (ensurePeered) — Before any outbound handshake, ensurePeered(origin) in federationPeering.ts refuses with { status: 'rejected', error: 'Local admin must resolve…' } if an inbound peer_approval_requests row exists for that origin. The guard query is narrowed to direction='inbound' (commit 0d3d087) so the gate's own outbound queue rows do not falsely block their own approval path. This blocks the auto-reconnect trigger: without the guard, any code path calling ensurePeered (e.g., the silent reconnect in stores/instanceStore.ts) could initiate a fresh outbound handshake to a peer that has a pending inbound approval request. The legitimate approve flow (POST /api/federation/approval-requests/:id/approve) does NOT call ensurePeered — it deletes the approval-request row and does its own direct fetch to /peer/accept — so the guard does not block legitimate approvals.

Approval Token Verification

The pre-handshake guard above closes the most reliable trigger but cannot prevent every adversarial code path on a remote side from sending an inbound /peer/accept against a row in awaiting_approval. To close that broader class, the protocol adds a cryptographic single-use approval token verified on the receiver before promoting awaiting_approval → active. Spec: docs/superpowers/specs/2026-04-26-peer-approval-token.md.

Issuance. When /peer/accept returns 202 (queue-as-approval-request, autoAcceptPeering=0), the server generates a 32-byte random hex token via crypto.randomBytes(32).toString('hex'), stores it on the new peer_approval_requests.approval_token column, and returns it in the 202 body as { queued: true, message, approvalToken }.

Storage on the initiator. When the initiator's outbound /peer/accept (from performHandshake, /peer/initiate, or /approve) receives 202, it parses approvalToken from the response body and stores it on the local federation_peers.approval_token column alongside status='awaiting_approval'.

Forwarding from /approve. When the local admin approves an inbound queued request, the /approve endpoint reads approvalToken from the queued peer_approval_requests row and includes it in its outbound /peer/accept body. No other code path forwards a token — the security property is "only /approve reads the stored token from the DB."

Verification on the initiator's /peer/accept handler. When an inbound request matches an existing awaiting_approval peer row, the handler verifies existing.approval_token === request.body.approvalToken (length-checked, plain === — see spec §3.1 on why constant-time comparison isn't required). On match, the row is promoted to active, the stored token is cleared (approval_token=NULL), and any stale peer_approval_requests row for the origin is deleted. On mismatch (or missing token), behavior depends on the receiver's autoAcceptPeering:

• autoAcceptPeering=0 → queueApprovalRequest() is invoked: a new peer_approval_requests row is upserted with a fresh token, returns 202. The existing awaiting_approval row is left untouched. No bypass.
• autoAcceptPeering=1 → fallback promote to active (no security regression vs. prior behavior — autoAccept=1 would accept any inbound /peer/accept regardless).

Single-use lifecycle. The token is consumed on first successful match: cleared from federation_peers.approval_token at the receiver's promote step, cleared from the initiator's federation_peers.approval_token when its own outbound returns 200, and deleted along with the approval-request row when /approve completes. This bounds replay of a leaked token (DB snapshot, log capture) to the period before the legitimate /approve runs.

Backward compatibility. Both schema columns are nullable. Older peers that don't include approvalToken in the request body or 202 response result in null storage; the receiver's verification then falls through the autoAcceptPeering gate. Existing active peers and existing awaiting_approval rows in production at upgrade time are unaffected — the verification only runs on the receiver's awaiting_approval branch. Stalled legacy awaiting_approval rows (no stored token) cannot complete via inbound /peer/accept from a legacy initiator unless the receiver is autoAccept=1; admins should re-initiate them through the standard flow if needed.

What this does NOT defend against — a remote operator running custom code with full DB access can read their stored token and forge /peer/accept. That is the inherent trust radius of federation peering. The threat model is bug-prone code paths (auto-reconnect, voice-call peering races, future ensurePeered callers) on otherwise-honest peers, not adversarial operators. Sender-side outbound gating for autoAcceptPeering=0 was tracked as the "Direction C" follow-up and is now closed by the Outbound Peering Gate below (spec docs/superpowers/specs/2026-04-26-outbound-peering-gate-design.md, commits ac2565f..c795d72).

Outbound Peering Gate

The receiver-side trust class is closed by the approval-token mechanism above. The sender-side trust class — bug-prone or incidental code paths on the initiator that drag the local instance into peering relationships without local admin consent — is closed by a centralized gate inside ensurePeered. After this change, autoAcceptPeering=0 is symmetric: BOTH inbound and outbound new-peer establishment require local admin approval. Existing active peers keep working unchanged.

Spec: docs/superpowers/specs/2026-04-26-outbound-peering-gate-design.md.

Gate location. ensurePeered (utils/federationPeering.ts) is the single chokepoint every outbound new-peer attempt funnels through (friend-add, /peer/ensure, sendCallRelay's no-active-peer branch, future callers). The gate runs ONLY when no federation_peers row exists for the origin. If any peer row exists in any status (active, pending, awaiting_approval, rejected, revoked, unreachable, needs_attention), the gate is a no-op — the existing branches in ensurePeered handle the row as today. This matches the threat model: the worry is automated paths creating new peerings without admin consent.

Required caller intent. Every call site MUST pass an explicit EnsurePeeredCallerIntent (declared in packages/shared/src/types.ts). The argument is required at the type level so a future caller cannot silently fall through to system behavior:

export type PeeringTriggerReason = 'friend_add' | 'space_join' | 'direct_message';

export type EnsurePeeredCallerIntent =
  | { kind: 'user_action'; userId: string; reason: PeeringTriggerReason; target: string }
  | { kind: 'system' };

export type EnsurePeeredResult =
  | { status: 'active'; peerId: string }
  | { status: 'rejected'; error: string }
  | { status: 'failed'; error: string }
  | { status: 'pending'; error: string }
  | { status: 'admin_required'; error: string };  // ← new variant

(Type was originally drafted as TriggerReason and renamed to PeeringTriggerReason in commit f6487a2 to disambiguate from unrelated outbox/voice trigger enums.)

Gate-but-don't-queue split. When the gate fires (no peer row + autoAcceptPeering=0), behavior diverges on intent:

• intent.kind === 'user_action' — upsert an outbound peer_approval_requests row keyed on (origin, direction='outbound'), upsert a peer_approval_subscribers row keyed on (request_id, user_id, trigger_reason, trigger_target), broadcast federation_approval_request_received to admins, broadcast peering_subscription_changed to the requesting user, return { status: 'admin_required', error: 'Awaiting your admin\'s approval to initiate peering' }. The user's request is admin-approvable but no traffic reaches the wire yet.
• intent.kind === 'system' — no DB writes, no admin broadcast, no admin-visible queue clutter; return { status: 'admin_required', error: 'Outbound peering requires admin approval on this instance' }. A stale outbox event or dead voice-call has no admin remedy worth surfacing.

'admin_required' result semantics. Each user-action caller maps the new variant to its own surface (e.g. friend-add returns 409 peer_pending_local_admin; sendCallRelay returns peer_admin_required and threads through the existing exhaustive CallRelayFailureReason switch). See social.md §6 outbound flow for the friend-add error mapping.

Lifecycle (centralized cleanup on onPeerActivated). The most important correctness invariant: outbound subscriber cleanup hangs off status transition to active (onPeerActivated), NOT off the local admin's approve-action handler. This holds across every activation path:

• queue approval (/api/federation/approval-requests/:id/approve 200 branch)
• admin-direct (/peer/initiate)
• autoAccept=1 remote (/peer/accept 200 from a remote that auto-accepts)
• mutual-token approval (the receiver's /peer/accept verifying approvalToken and promoting awaiting_approval → active)

When federation_peers.status transitions to active, onPeerActivated (utils/federationPeerActivation.ts) calls fanoutOutboundSubscribers(origin): it queries the outbound peer_approval_requests row for the activated origin, inserts a kind='approved' row in peer_approval_notifications for each subscriber, broadcasts peering_notification_received per subscriber, then deletes the parent peer_approval_requests row (cascade clears subscribers). No per-action code knows about subscribers; the queue-approval handler does NOT duplicate this cleanup — it just performs the handshake and lets the resulting status transition trigger fanout.

The reason cleanup must NOT live in the approve-action handler: when the remote also has autoAcceptPeering=0, local admin approval transitions our peer row to awaiting_approval, not active. Subscribers must remain queued until full activation completes (which may be days later when the remote admin approves). Wiring cleanup to the approve-action handler instead would clear subscribers prematurely; users would retry against an awaiting_approval peer and hit peer_pending_approval 409, confused.

The other lifecycle exits write notifications and cascade-delete the parent at the trigger site:

• Admin denies an outbound request (POST /api/federation/approval-requests/:id/deny, direction='outbound' branch) — fans out kind='denied' notifications to subscribers, then deletes the parent (cascade clears subscribers). No remote network call — the remote never knew we were considering this peer. federation_peers_changed is broadcast to admins so the queue UI refreshes.
• Last-subscriber cancel (DELETE /api/federation/peering-subscriptions/:id) — deletes the subscriber row; if it was the last subscriber for the parent, cascade-deletes the parent. No notification created (the user took the action; they know).
• Parent-row expiry (storageJanitor.ts outbound branch) — fans out kind='expired' notifications to subscribers before deleting the parent. Inbound expiry behavior is preserved unchanged from pre-branch: the janitor still sends a signed /peer/denied to the inbound origin and only deletes the row on success (the Task 9 first-pass mistakenly removed this; commit c4e7438 restored it).

No status column on subscribers. Existence = waiting; deletion = resolved (with the resolution mode encoded in the notification row created at deletion time, except for the canceller path).

Gate composition with the trust-guard. The pre-handshake trust-guard ("inbound peer_approval_requests exists → refuse outbound with 'rejected'") is preserved unchanged and runs after the gate. The trust-guard query was narrowed to direction='inbound' so the gate's own outbound queue rows don't false-positive block their own approval path. The two layers compose cleanly: the gate fires earlier when there's no peer row at all; the trust-guard fires later when an inbound row exists.

Caller summary (intent each call site declares):

Site	Intent	On 'admin_required'
routes/social.ts (friend-add)	user_action reason friend_add target name@domain	409 peer_pending_local_admin
routes/federation.ts (/peer/ensure)	user_action reason friend_add (default; client-supplied reason is ignored today)	response peeringStatus: 'admin_required'
utils/federationOutbox.ts (sendCallRelay no-active-peer)	system	CallRelayFailureReason.peer_admin_required
utils/federationWorker.ts (resolvePendingPeers)	n/a	operates only on already-existing pending rows; gate is unreachable from this path

Admin-initiated paths (/peer/initiate, /approve) do NOT call ensurePeered. They issue their own fetch and run their own activation logic. The gate does not affect admin power; admins retain full ability to pre-peer or approve outbound regardless of the setting.

Admin Endpoints

Endpoint	Method	Auth	Purpose
/api/federation/peer/initiate	POST	JWT + admin	Start peering handshake
/api/federation/peer/accept	POST	None (rate-limited)	Accept incoming handshake
/api/federation/peer/ensure	POST	JWT (any user), rate-limited 3/15min/user	Trigger auto-peering to a remote instance
/api/federation/peers	GET	JWT + admin	List all peers (secret excluded)
/api/federation/peers/:id	PATCH	JWT + admin	Update peer settings (auto-rotation interval)
/api/federation/peers/:id	DELETE	JWT + admin	Revoke peer, purge outbox
/api/federation/peers/:id/permanent	DELETE	JWT + admin	Hard-delete revoked peer record
/api/federation/peers/:id/reset	POST	JWT + admin	Delete peer record (cascade-deletes outbox). Only admissible in needs_attention state.
/api/federation/peers/:id/rotate	POST	JWT + admin	Trigger immediate secret rotation
/api/federation/peers/:id/recheck	POST	JWT + admin	Run an immediate reachability probe on an unreachable peer; 400 unless status='unreachable'; 200 { recovered, status }
/api/federation/approval-requests	GET	JWT + admin	List pending peering approval requests (inbound + outbound). Outbound rows include subscribers: ApprovalRequestSubscriberSummary[] (possibly empty). Inbound rows omit subscribers. Each row carries direction: 'inbound' | 'outbound'.
/api/federation/approval-requests/:id/approve	POST	JWT + admin	Approve request — direction-branched (see Approval flow above)
/api/federation/approval-requests/:id/deny	POST	JWT + admin	Deny request — direction-branched (see Denial flow above)

POST /api/federation/peer/ensure — Wraps ensurePeered(). Accepts { remoteOrigin: string } in body. Returns { peeringStatus, peerId?, error? } where peeringStatus is one of active, pending, awaiting_approval, rejected, unreachable, or revoked.

S2S Endpoints

Endpoint	Method	Auth	Purpose
/api/federation/peer/rotate	POST	HMAC	Accept secret rotation from peer
/api/federation/peer/denied	POST	HMAC	Receive denial notification for awaiting_approval peer
/api/federation/identity	DELETE	HMAC	Delete federated user identity (soft/full mode)
/api/federation/users/lookup	POST	HMAC, rate-limited 60/min/peer	Resolve a username on this instance to (homeUserId, profile snapshot) for cross-instance friend-request originators
/api/federation/epoch	POST	HMAC (signed request and signed response)	Return this instance's persistent epoch { instanceId }; populates a peer's trusted epoch baseline (peer_instance_id)

S2S Epoch Refresh (POST /api/federation/epoch)

HMAC-authenticated in both directions: the request is signed (only a peer holding the shared secret may call it — unknown/revoked peers → 403, bad signature → 401, missing headers → 400) and the response body { instanceId } is HMAC-signed with the same secret (X-Federation-Signature/Timestamp/Nonce response headers). The caller (fetchPeerEpoch(peer) in utils/federationEpoch.ts) verifies that response signature with the same secret before trusting the value, then writes it to federation_peers.peer_instance_id. Response-signing (not TLS-only) is deliberate: a poisoned baseline could drive a spurious data-heal on a live peer, so the newly-trusted epoch is authenticated (design §9). fetchPeerEpoch fails safe — a 404 from a not-yet-upgraded peer, an absent/invalid response signature, or a network/timeout error all return null (10s timeout via AbortSignal.timeout); the caller treats null as "retry on the next tick," never as an error to surface. This is the deterministic populator of the epoch baseline (the bounded periodic epoch-refresh, design §3.2), independent of organic relay traffic.

Deterministic epoch-refresh driver (refreshPeerEpochs() in utils/federationEpoch.ts). Selects every active peer whose peer_instance_id IS NULL, calls fetchPeerEpoch(peer) once each, and on a non-null result writes the epoch via UPDATE ... SET peer_instance_id WHERE id = ? AND peer_instance_id IS NULL. The trailing IS NULL guard makes it populate-if-null only — it can never overwrite a baseline another path (relay envelope, handshake) already established — and makes it self-terminating: once a peer's peer_instance_id is set, the IS NULL filter excludes it, so it is never fetched again. A null from fetchPeerEpoch (404 / bad-sig / network) is a benign continue with no error log-spam, retried next tick. Wired into the federation worker in two places: once at startFederationWorkers() startup and once at the end of processHealthCheckTick() (the existing 15-minute health-check tick), both as refreshPeerEpochs().catch(() => {}). This guarantees the trusted baseline is populated within one refresh cycle of an upgrade, independent of user/relay activity — the load-bearing populator that relay-only population cannot cover for idle peers.

Reset Detection (markPeerReset — utils/federationReset.ts)

The epoch is a detection signal only, never an authorization signal. When a peer behind a known origin advertises an epoch that differs from the trusted baseline (peer_instance_id), the instance was wiped and a new incarnation stood up on the same domain. markPeerReset(peerId, origin, deadEpoch, observedEpoch) routes the peer for admin attention and snapshots the dead incarnation — but performs NO rekey, NO tombstone, NO handle change, NO content deletion. The actual data heal fires only later, from onPeerActivated after an admin-authenticated re-peer (design §6). Because detection grants no capability and destroys nothing, it is safe to fire on an unauthenticated signal: the worst a spoofed detection can do is flag a peer for admin review (admin-reversible nuisance).

In a single transaction, markPeerReset:

1. Sets the peer status='needs_attention', needs_attention_reason='peer_reset_detected', and observed_peer_instance_id=observedEpoch. peer_instance_id (the trusted baseline) and hmac_secret are left untouched — an unauthenticated observation never rekeys trust; the observed-but-untrusted epoch lives only in observed_peer_instance_id.
2. Snapshots the dead incarnation: sets users.federation_heal_pending = 1 for every non-deleted user whose home_instance matches the origin. The match keys on extractDomain(origin) (bare domain, the canonical home_instance form) and defensively also matches the https:///http://-prefixed forms so any legacy full-URL straggler is caught (homeInstanceMatch()). Any stub created after detection (e.g. a friend-add reaching the new incarnation directly) is un-flagged and survives the heal.
3. Journals the dead incarnation durably by upserting a federation_reset_events row keyed by origin: { dead_epoch=deadEpoch, new_epoch=NULL, detected_at, resolved_at=NULL, stub_count, orphaned_account_count }. stub_count counts flagged pure S2S stubs (password_hash = '!federation-replicated'); orphaned_account_count counts flagged real accounts. This row survives the peer-row deletion that Re-peer performs, preserving dead_epoch for the false-positive guard (design §6.1) and the admin surface.
4. After the transaction, broadcasts federation_peers_changed and federation_peer_reset_detected {origin} to admins.

Idempotent / double-reset: if an unresolved federation_reset_events row already exists for the origin (the peer reset again before an admin resolved the first), the original dead_epoch and detected_at are preserved (that is the incarnation whose users are already snapshotted) — only the summary counts are refreshed. dead_epoch is never overwritten on an unresolved row. A prior resolved reset starts a fresh journal entry.

Detection sources (all three wired in this feature):

• Inbound handshake — /peer/accept landing on an active/needs_attention row (routes/federation.ts): before the idempotent-200 return, if (reqInstanceId && existing.peerInstanceId && reqInstanceId !== existing.peerInstanceId) markPeerReset(...). The guard still returns 200 and still does not rekey — the anti-hijack property is preserved verbatim; detection is layered on top.
• Reachability probe (unreachable peers) — probePeerReachable() (utils/federationRecovery.ts) now parses instanceId from the /api/instance/info response (returning { reachable, instanceId }; a missing/unparseable epoch is null, never an error). The shared decision helper recoverOrDetectReset(peer, result) — used by both the background recovery tick (processRecoveryTick) and the manual recheck endpoint — routes to markPeerReset (returning 'reset_detected') when the peer has a non-null peer_instance_id and the probed epoch differs, and does NOT call markPeerRecovered. Rationale: a genuinely reset peer's HMAC secret is desynced, so flipping it back to active via a reachability probe would resume relay against a dead secret. Only when the probed epoch matches the baseline (or the baseline is null / epoch unknown) does the normal recovery-to-active path run. The manual recheck endpoint returns { recovered: false, status: 'needs_attention' } on 'reset_detected'.
• needs_attention-peer reset probe — closes design §4.1's remaining sub-case. A reset peer can reach needs_attention via the auth-failure path — its HTTP is up but returns 401/403 because the new incarnation has no peer row for us, so consecutive_auth_failures crosses AUTH_FAILURE_THRESHOLD — without ever transitioning through unreachable. The unreachable-only 5-second recovery probe therefore never observes its epoch change, so no journal is created; a later manual Re-peer would then run healResetIncarnation with no journal row → no heal → the split-brain persists. The shared per-peer unit is detectResetForPeer(peer) (utils/federationRecovery.ts): for a peer with a non-null baseline it runs one probePeerReachable (/instance/info GET) and calls markPeerReset on an observed epoch mismatch. It is invoked from three places, so detection latency is near-zero rather than up to a full health-check cycle:
  1. Event-driven, at the transition — the instant the outbox worker moves a peer to needs_attention on the auth-failure threshold (federationWorker.ts), it fires detectResetForPeer for that peer (fire-and-forget). This is the common live case: the moment the connection is declared broken, the epoch is checked and "Re-peer & heal" surfaces immediately.
  2. Worker-startup sweep — startFederationWorkers() runs detectResetOnNeedsAttentionPeers() once on boot, catching any peer already parked in needs_attention (reset while this instance was down, or transitioned before this probe shipped).
  3. 15-minute health-tick backstop — detectResetOnNeedsAttentionPeers() also still runs at the end of processHealthCheckTick as the periodic safety net. It selects peers with status='needs_attention' AND peer_instance_id IS NOT NULL AND needs_attention_reason not already peer_reset_detected (those already carry a journal) and calls detectResetForPeer on each. Detection only: unlike recoverOrDetectReset, none of these ever flips a needs_attention peer to active (a match / unknown / unreachable result is a pure no-op) — that peer's secret is desynced and only an admin-authenticated re-peer restores trust. Neither peer_instance_id nor hmac_secret is touched.

Legacy peers advertise no epoch (instanceId null), so detection requires a non-null observed epoch differing from a non-null stored baseline — legacy peers never trigger it, and the existing auth_failures → needs_attention → manual Reset path continues unchanged for them.

Limbo-window user error (peer_reset_pending)

Between reset detection (markPeerReset) and the admin's one-click Re-peer, the stale identity graph still exists and no heal has run (design §5.3). During this window a user re-adding a same-name friend, or creating a DM to that origin, would otherwise hit a confusing already_friends (stale friendship bound to the dead incarnation) or peer_rejected (the peer now sits in needs_attention, tripping ensurePeered). Both user-facing hot paths short-circuit with a clearer 409 { error: 'peer_reset_pending' }:

• Friend-add (social.ts POST /api/social/requests, federated branch): after resolveOriginFromHostname yields peerOrigin and before the peering/lookup/already_friends checks.
• DM-create (dm.ts POST /api/dm, homeUserId + homeInstance branch): before stub creation, so no un-flagged stub is left behind. The target origin is resolved with resolveOriginFromHostname(new URL(canonicalizeHomeInstance(homeInstance)).host).

Both perform an O(1) point lookup on the federation_reset_events origin PRIMARY KEY (origin = peerOrigin AND resolved_at IS NULL). peerOrigin (from resolveOriginFromHostname, which returns the stored federation_peers.origin verbatim) is exactly the string markPeerReset journals, so the query is a single indexed hit/miss. The guard only short-circuits when an unresolved row exists; the common case — no reset in progress — is one indexed miss and the normal path proceeds byte-for-byte unchanged. Once the admin re-peers and healResetIncarnation resolves the journal (resolved_at set), the guard stops firing and the freshly-clean graph accepts the re-add.

Data Self-Heal (healResetIncarnation — utils/federationReset.ts)

Detection (markPeerReset) only snapshots + journals + notifies; it destroys nothing. The actual heal is healResetIncarnation(origin, newEpoch, reason), fired from onPeerActivated (utils/federationPeerActivation.ts) after an admin-authenticated re-peer, keyed to the confirmed epoch change (design §6). It runs before the mutation-log re-sync in onPeerActivated so re-sync repopulates onto a clean slate, and it runs outside any transaction (tombstoneUser opens its own; better-sqlite3 throws on a nested BEGIN).

Two mandatory guards, in order:

1. Reason gate. onPeerActivated fires on non-handshake paths too. Only genuine re-handshake reasons carry a freshly-exchanged, trustworthy epoch. healResetIncarnation returns immediately unless reason is in the allow-list HANDSHAKE_ACTIVATION_REASONS (typed ReadonlySet<PeerActivationReason>): initiate_accepted, accept_new, accept_pending, accept_rejected_override, accept_awaiting_approval, accept_awaiting_approval_fallback, approval_handshake, ensure_peered. The two EXCLUDED reasons — health_check_recovery (reachability flip in markPeerRecovered) and startup_bootstrap (boot re-scan) — flip a peer to active without a handshake, so their baseline is stale (still equals the journaled dead_epoch). Without the gate they would hit the deadEpoch === newEpoch false-alarm branch and silently resolve the journal + clear the flags WITHOUT healing, permanently burying the bug. Gated out, they leave the journal fully intact for a later genuine re-handshake to heal.

2. Epoch comparison (false-positive guard). For a gated-in reason, look up the UNRESOLVED federation_reset_events row for the origin (none → return):

   • journal.dead_epoch === newEpoch — the re-peer confirmed the SAME incarnation (spurious/spoofed detection, or an admin re-peer to a never-reset live peer). NO tombstone — the user-level snapshot flags alone must never authorize destruction; only a confirmed epoch change does. Clears federation_heal_pending for the origin and resolves the journal (new_epoch, resolved_at).
   • journal.dead_epoch !== newEpoch — a GENUINE new incarnation. Soft-tombstones the flagged pure stubs only, then clears their flags and resolves the journal.

Soft-tombstone (pure stubs only). For every user that is federation_heal_pending = 1 AND password_hash = '!federation-replicated' (pure S2S stub sentinel) AND matches the origin (homeInstanceMatch), calls tombstoneUser(uid, { purgeContent: false }). The purgeContent: false is non-negotiable — the default (true) irreversibly deletes this box's reactions and authored space messages, violating the invariant that a remote's reset never destroys our non-re-syncable content. The soft tombstone clears exactly the relationship rows that cause the bug (friends, friend_requests, group dm_members, …) so stale friendships/DMs clear and re-adds work. A 1-on-1 dm_members row is now KEPT and anonymized so the survivor retains a read-only "Deleted User" thread — see dm-system.md § "DM Tombstone Semantics". Flags are then cleared keyed by the stub id list (not by re-querying the sentinel — tombstoneUser has already randomized password_hash).

Live co-member update. Around each stub, the heal loop now broadcasts a sanitized user_updated so survivors' clients flip the partner to "Deleted User" without a reload — aligning this path with the other three deletion callers (admin/self/identity-delete). For each stub it captures collectDeletionBroadcastTargets(stub.id).targetUserIds before tombstoneUser (which deletes the DM/friend/space rows that set is derived from), then re-reads the tombstoned row and calls connectionManager.sendToUser(targetId, { type: 'user_updated', user: sanitizeUser(deletedRow) }) for each target.

Real federated accounts — detach (design §6.3b, revised by the 2026-07-02 orphaned-account-detach spec). A flagged user that is NOT a stub (password_hash != '!federation-replicated') carries real, non-re-syncable local content and is never auto-tombstoned. In the genuine-reset branch, after the stub soft-tombstone loop, healResetIncarnation calls quarantineOrphanedAccounts(origin):

• For every flagged real account (federation_heal_pending = 1, non-stub, isDeleted = 0, homeInstanceMatch): set federation_home_orphaned = 1 and clear federation_heal_pending. That is all — this is a flag-only detach, not a freeze.
• federation_home_orphaned = 1 means "DETACHED / sovereign local account," not "frozen." The account was cut loose from its (now-reset) home instance and operates as a purely local account from here on: it logs in with its local password (auth.ts no longer blocks a detached account before password verify — only the self-heal branch is permanently disabled for it, see auth.md §4), and it gains local profile edit + local change-password (users.ts). There is no login freeze.
• No rename. The username is preserved — first-come-first-served on this instance. There is no !orphaned:{uid}@{domain} handle-freeing and no space-owner special case: all real accounts are treated uniformly and owners simply keep managing their spaces.
• What closes the post-re-peer hijack is NOT a login freeze. It is the combination of (a) the login self-heal being permanently disabled for detached accounts (auth.ts returns 401 in the failed-local-password branch without contacting the home domain — a new incarnation can never re-hash its way in) and (b) the S2S identity-binding guards, which exclude detached rows on every domain-keyed surface: findFederatedUser tier-2 (federation_home_orphaned = 0 predicate, ~line 3522), the S2S profile_update handler (accept-and-skip, ~line 6162), the S2S presence_update handler (accept-and-skip, after the domain-collision check), the hydrateReplicatedUserProfile fill-empty path (no-op early return alongside the native-user skip), and the S2S DELETE /api/federation/identity guard (idempotent 200, no deletion, ~line 2357). Every domain-keyed mutation that a tier-1 (homeUserId) hit can reach is guarded at its own site — a tier-1 hit on a detached row is a legitimate historical read, but no write is applied. See "S2S Identity Deletion" above and design §4.3.
• Content (space messages, memberships, reactions) and usernames are preserved in all cases. No user_updated broadcast is emitted (nothing visible changes). The returned count refreshes the journal's orphaned_account_count.

Login self-heal epoch guard (design §6.3a). The federated password self-heal (auth.ts §4) gates re-hashing on the home instance's current epoch, read via the authenticated fetchPeerEpoch(peer) (HMAC-signed both ways): no baseline on record → allow (legacy); baseline differs from the fetched epoch → refuse; epoch can't be determined (fetchPeerEpoch null — 404/unreachable/bad-sig/desynced secret) → fail closed/refuse; match → allow. This closes the pre-re-peer hijack (a reset home accepting a new same-name user's password during the undetected-reset window). The post-re-peer window is closed by detach: once an account is detached, its self-heal path is permanently disabled and the S2S binding guards exclude it (above), so the new incarnation has zero influence over it. Full three-way in auth.md §4.

Reset-events admin surface (GET /api/federation/reset-events). Admin-only. Returns the durable federation_reset_events journal (each event carrying a nullable acknowledgedAt) joined with each origin's current detached real accounts (federation_home_orphaned = 1, homeInstanceMatch), each with ownedSpaces, spaceMemberCount, and authored-messageCount for disposition. Response type FederationResetEventsResponse ({ events: FederationResetEvent[] }, each event carrying orphanedAccounts: FederationOrphanedAccount[]). The endpoint returns all events (including acknowledged ones, for audit); the client filters to acknowledgedAt === null. Disposition actions — one-click Re-peer (/peers/:id/reset → /peer/initiate), full-purge Remove (DELETE /api/admin/users/:id, owns-spaces → transfer first) for genuinely-abandoned detached accounts, and a non-destructive Dismiss (POST /api/federation/reset-events/acknowledge with { origin }, idempotent — sets acknowledged_at) that hides the card without touching the accounts (detached accounts keep working locally). See admin.md "FederationPanel" and client-federation.md §8.

needsAttentionReason on the peer API. GET /api/federation/peers returns needsAttentionReason: 'auth_failures' | 'peer_reset_detected' | 'repeer_incomplete' | null per peer, so the admin UI distinguishes a reset-detected peer (persistent Reset-cleanup banner + one-click Re-peer) from a generic auth-failure peer (plain "Reset Peering") and from a peer whose Re-peer could not be cryptographically verified (repeer_incomplete — see "Trust re-establishment contract"). repeer_incomplete is a nullable-TEXT value only; no schema migration.

Trust re-establishment contract

The peering handshake must never report success when trust was not actually re-established — a false success re-creates the permanent HMAC desync ("split-brain") the epoch feature exists to prevent. Two changes make the recovery handshake honest end-to-end, and the anti-hijack guard is unchanged (an unauthenticated /peer/accept never adopts a caller's secret over an existing active/needs_attention row; no path auto-rekeys).

1. Responder honest refusal (POST /api/federation/peer/accept). When a peer row already exists in active OR needs_attention, the endpoint returns 409 { accepted: false, code: 'PEER_EXISTS_RESET_REQUIRED', error, instanceName, instanceId } instead of the old false 200 { accepted: true }. It still does not adopt the caller's hmac_secret (identical guard behavior — only the reported status/body changed), and the epoch-mismatch markPeerReset detection still fires before the return. Legacy initiators that only read response.ok now fail loudly instead of silently desyncing; new initiators special-case the code.

2. Initiator verify-before-activate (POST /api/federation/peer/initiate).

• On remote 409 PEER_EXISTS_RESET_REQUIRED: the initiator DELETES its own pending row (keeping its slot clean so the remote's own Re-peer can later land on a fresh responder slot) and returns 409 { code: 'PEER_EXISTS_RESET_REQUIRED', error }.
• On remote 200: the initiator performs a signed fetchPeerEpoch (POST /api/federation/epoch) round-trip before activating. Null (secret doesn't authenticate / /epoch absent / unreachable) → the peer is parked in needs_attention with needs_attention_reason='repeer_incomplete' and the response is 200 { peer, verified: false }. Success → the peer activates (storing the cryptographically-verified epoch as peer_instance_id) and the response is 200 { peer, verified: true }.

Recovery flows.

• Common reset case (one click from the survivor). When the reset box makes contact, the survivor's detection (markPeerReset) moves the survivor's row to needs_attention. The survivor's Re-peer (reset local row → /peer/initiate) then lands on the reset box's clean responder slot → both sides re-key and the initiator verifies via fetchPeerEpoch → both active with a matching secret. If the reset box initiated first, the responder's new 409 refusal (change 1) means the reset box deletes its pending row rather than false-activating, so the survivor's subsequent Re-peer still finds a clean slot.
• Bidirectional-stale case. If both sides still hold a conflicting row, each side that holds one must reset once — the honest 409 / verified:false reporting tells the admin exactly that ("the remote still holds stale peering for you; its admin must reset their side, then Re-peer again"), instead of falsely reporting success on a dead peering.

Backward-compat honesty (limitation, stated explicitly). A genuinely pre-Phase-1 peer whose /api/federation/epoch returns 404 cannot be cryptographically verified, so /peer/initiate parks it in needs_attention (repeer_incomplete) rather than activating. This is intentional fail-safe: a legacy responder that returns 200 without adopting the secret is indistinguishable from a healthy fresh peering, so we refuse to trust it blind. All current Backspace instances ship /epoch (Phase 1), so this only affects truly pre-Phase-1 peers — it is an honest limitation, not a silent behavior change for healthy peers.

Handshake sourceOrigin honors PUBLIC_ORIGIN. resolveLocalOrigin() (routes/federation.ts) now delegates to getOurOrigin(), so the origin advertised in the handshake sourceOrigin is byte-identical to the X-Federation-Origin used for all authenticated S2S requests. Previously it used https://${DOMAIN} and ignored PUBLIC_ORIGIN, which silently desynced the responder's peer-row key from the auth origin on any instance where PUBLIC_ORIGIN != https://DOMAIN — producing permanent 403 Not peered. See "Public Origin Override" below.

Verification harness. The self-contained two-instance integration harness (packages/server/test/helpers/realHandshake.ts + packages/server/test/federation-handshake-desync.test.ts) exercises the REAL cross-instance handshake — actual /peer/initiate→/peer/accept, the signed /epoch health probe (s2sHealthy), and simulateReset — using ephemeral in-process instances (each with PUBLIC_ORIGIN set to its http://127.0.0.1:<port> transport URL, its own temp DB and generated secrets, no live host). Case #1 is the clean-handshake control; #2 gates the responder-refusal fix (BUG-1); #4 gates one-click Re-peer recovery (BUG-2).

S2S Identity Deletion (DELETE /api/federation/identity)

Allows a home instance to remove a user's replicated identity from a remote instance.

Request body:

{ "homeUserId": "<string>", "homeInstance": "<string>", "mode": "soft" | "full" }

Behavior:

• Attribution guard: Rejects with 403 if the user's homeInstance doesn't match the X-Federation-Origin of the signing peer. Prevents one instance from deleting another instance's users.
• Detached-account guard: After the attribution guard, if the resolved user has federation_home_orphaned = 1 (detached — its home domain was reset and it is now a sovereign local account, see §4.2/§4.3 of the orphaned-account-detach design), returns an idempotent 200 { success: true } without deleting. A new incarnation on the reset domain must never delete an established account by replaying its old homeUserId.
• Idempotent: Returns { success: true } for already-deleted or nonexistent users (no error).
• Owned spaces check: Returns 409 with { ownedSpaces: string[] } if the user owns any spaces on the remote. The user must transfer or delete those spaces before identity removal proceeds.
• Mode "soft": Calls tombstoneUser(uid, { purgeContent: false }) — anonymizes the user row and removes the user from spaces, friends, DM membership (dm_members), and read-states. The purgeContent: false flag skips only reactions, dm_reactions, and the user's space messages (with attachments + embeds); DM membership cleanup and orphaned-DM purge always run in both modes (per userDeletion.ts:121-126, 169-202) because zero-member DM channels are unreachable garbage regardless of authorship retention.
• Mode "full": Calls tombstoneUser(uid, { purgeContent: true }) — full tombstone including reactions and orphaned DM cleanup.
• Post-deletion: Broadcasts member_left WS events for all spaces the user belonged to before removal.

S2S User Lookup (POST /api/federation/users/lookup)

HMAC-authenticated. Rate-limited to 60 requests/minute per peer. Used by the cross-instance friend-add flow to resolve a username on this instance before the sender's home server queues a friend_request_create event (see social.md §6 outbound flow).

Request body: { username: string } — server trims and lowercases before lookup.

Response 200: { found: true, user: { homeUserId, username, profile: { displayName, avatar, avatarColor, banner, bio } } } — returned for native, non-deleted users regardless of their discoverable setting.

Response 404: { found: false, code: 'user_not_found' } — returned for tombstoned users (isDeleted=1), replicated stubs (homeInstance IS NOT NULL), or unknown usernames.

Response 400: Malformed input (missing, non-string, or empty-after-trim username).

Response 429: Per-peer rate limit (60/min) exceeded; Retry-After header set.

Filter invariant: discoverable is NOT consulted — exact-handle resolution must work for opted-out users. This mirrors the Direct-Add invariant from social.md commits 69d430b/c60ecc5/189cac4: discovery surfaces only opted-in users, but once you have a handle you can always send a request.

WebSocket Events (Peering)

These S→C events are pushed to the acting user's connected clients by the federation subsystem.

Event	Pushed when	Payload
federation_peer_rejected	Outbox worker receives 403 PEERING_REQUIRES_APPROVAL from a remote instance during auto-peering	{ peerId: string, origin: string }
federation_peer_active	A previously rejected peer transitions to active (e.g., via manual peer/initiate or incoming peer/accept)	{ peerId: string, origin: string }
federation_peer_reset_detected	A peer's advertised instance epoch differs from the trusted baseline (wipe-and-reinstall on the same domain) — emitted by markPeerReset after routing the peer to needs_attention	{ origin: string } (admin-only, via sendToAdmins)

---

2. HMAC Request Authentication

Signing Format

HMAC-SHA256(secret, "${timestamp}.${requestBody}")

Where timestamp is Date.now() (Unix milliseconds) and requestBody is the JSON string.

HTTP Headers

Header	Format	Example
X-Federation-Signature	sha256=<hex>	sha256=a1b2c3...
X-Federation-Origin	Full URL	https://nova.ddns.net
X-Federation-Timestamp	Unix ms string	1711619400000
Content-Type	application/json	--

Verification (federationAuth.ts:verifySignature)

1. Validate inputs: reject empty/missing body, signature, or secret
2. Timestamp window: Math.abs(Date.now() - timestamp) <= maxAgeMs (default 15 minutes)
3. Recompute: HMAC-SHA256(secret, "${timestamp}.${body}")
4. Constant-time comparison: crypto.timingSafeEqual on hex-decoded buffers
5. Length check: mismatched buffer lengths are rejected before timingSafeEqual

Replay Attack Prevention

Two layers of replay protection:

1. Timestamp window: Requests older than 15 minutes are rejected (DEFAULT_MAX_AGE_MS).
2. Nonce: Each request includes a X-Federation-Nonce header (UUID v4). The nonce is included in the HMAC payload (${timestamp}.${nonce}.${body}) so it cannot be stripped. The receiver stores seen nonces in memory (keyed by peer origin, evicted after 15 min) and rejects duplicates with 409 Conflict.

Auto-ratchet: The nonceSupported column on federation_peers tracks whether a peer has ever sent a nonce. Once set to 1, nonce-less requests from that peer are permanently rejected (401). This allows graceful rollout — new peers get nonce enforcement automatically, legacy peers are warned in logs until they upgrade.

Inbound Verification Flow (POST /api/federation/relay)

1. parseFederationHeaders() extracts origin, timestamp, signature, and nonce from headers
2. Look up peer by origin in federation_peers -- must exist and be status = 'active'
3. Re-serialize request body to JSON: JSON.stringify(request.body)
4. verifySignature(bodyString, signature, peer.hmacSecret, timestamp, nonce) -- reject if false
5. Nonce enforcement: duplicate nonce → 409, missing nonce from ratcheted peer → 401, legacy peer → warn

Important: The body is re-serialized server-side. This means Fastify's JSON parsing and re-stringification must produce identical output to the sender's JSON.stringify. In practice this works because both sides use standard JSON.stringify with no custom replacers.

Relay-envelope epoch (fast-path baseline population, design §3.2). FederationRelayRequest carries sourceInstanceId?: string — the sender stamps its current epoch (getInstanceId()) when building the request in federationWorker.ts. Because the whole body is HMAC-verified above (step 4), a valid relay authentically carries the sender's current incarnation id. Immediately after the signature check passes (and only there — the authenticated boundary), the receiver runs populate-if-null: if (sourceInstanceId && peer.peerInstanceId IS NULL) UPDATE federation_peers SET peer_instance_id = <claimed> WHERE id = ? AND peer_instance_id IS NULL. This is the fast-path baseline populator — it fills the trusted epoch the instant organic traffic flows, usually before the deterministic 15-minute refreshPeerEpochs backstop fires. It never overwrites a non-null baseline: a differing incarnation implies a different HMAC secret that would have failed verification, so a valid relay can never carry an epoch differing from an established baseline. Runs independent of per-event processing and does not affect relay accept/reject. Backward-compatible: older peers omit sourceInstanceId → the update is skipped (no-op).

---

3. Identity Resolution

Functions

extractDomain(homeInstance) -- federation.ts

• Extracts bare domain from a homeInstance value (full URL or bare domain)
• "https://nova.ddns.net" → "nova.ddns.net", "nova.ddns.net" → "nova.ddns.net"
• Use when: Normalizing homeInstance for comparison or storage

findFederatedUser(homeUserId, homeInstance, db, hints?) -- federation.ts

• Three-tier lookup: homeUserId match → domain + username hint match → not found
• Tier 1: delegates to resolveLocalUser (fast path)
• Tier 2: uses extractDomain(homeInstance) + hints.username to match stubs created by the auth registration path (which may have a different homeUserId)
• Tier 2 excludes detached accounts (federation_home_orphaned = 1): a detached account is sovereign and must never be re-bound to the reset domain's new incarnation via username heuristics — that is exactly how a new same-name user would capture the established account. Tier 1 (homeUserId match) is deliberately NOT excluded: the new incarnation mints fresh homeUserIds, so a tier-1 hit on a detached row is a legitimate historical reference (e.g. an old group-DM attribution relayed by a third instance), not the new incarnation. Mutations are blocked at their own sites (profile_update handler, presence_update handler, hydrateReplicatedUserProfile fill-empty, S2S identity delete).
• Side-effect-free — does not modify any records
• When multiple candidates match in tier 2, prefers real accounts over stubs, then most profile data
• Use when: Read-only lookup that needs to find users created by either auth or relay path

resolveLocalUser(homeUserId, db) -- federation.ts

• Read-only lookup. Returns undefined if not found.
• Matches: (users.homeUserId = homeUserId) OR (users.id = homeUserId AND homeInstance IS NULL)
• Excludes deleted users (isDeleted = 0)
• When multiple candidates exist: prefers the one with homeUserId set (replicated stub) over a local ID match
• Use when: Optional lookups where null is acceptable (member_remove, reaction processing, friend_remove)

resolveOrCreateReplicatedUser(homeUserId, homeInstance, db, hints?) -- federation.ts

• Calls findFederatedUser first. If found, backfills homeUserId for future fast-path lookups and returns.
• Accepts optional hints: { username?: string | null } for tier-2 matching
• If not found, creates a stub with homeInstance normalized to bare domain via extractDomain
• Collision-safe: appends _1, _2, ..., _10 suffix if username exists; after 10 attempts, uses _<random hex>
• Use when: You MUST have a valid user ID. Always pass { username: profile?.username } when profile data is available.

hydrateReplicatedUserProfile(user, profile, db) -- federation.ts:2041

• Updates replicated stubs only (homeInstance must be set)
• Only updates null/empty fields (preserves manually-set local values)
• Exception: avatar/banner are overwritten if the current value is a bare filename (not an absolute URL)
• Resolves bare filenames to {homeInstance}/api/uploads/{filename} absolute URLs
• Sets displayName from profile.displayName || profile.username -- ensures federated users show a human-readable name instead of user@instance

Critical Rule

Any code path that sets ownerId, creates a dm_members row, or inserts a message MUST use resolveOrCreateReplicatedUser. Using resolveLocalUser with a ?? null fallback can cause data corruption (e.g., ownerId set to null for group DMs).

Origin Normalization

Two formats exist in the database:

Location	Format	Example
users.home_instance	Bare domain (normalized)	nova.ddns.net
federation_peers.origin	Full URL	https://nova.ddns.net
getOurOrigin() return	Full URL	https://orbit.ddns.net
resolveOrCreateReplicatedUser stores	Bare domain (normalized via extractDomain)	nova.ddns.net
Auth registration stores	Bare domain	nova.ddns.net

Both user creation paths now store bare domain. A self-healing migration in migrate.ts normalizes any existing full-URL homeInstance values to bare domain on startup.

Normalization pattern used in code:

const normalized = homeInstance.startsWith('http') ? homeInstance : `https://${homeInstance}`;

Locations where normalization is applied:

• getGroupDmTargetOrigins() (federationOutbox.ts:294) -- normalizes before comparing to ourOrigin
• dm.ts:655 -- isLocalMember broadcast filter checks both formats
• dm.ts:743 -- normalizes target homeInstance before peer origin comparison

Attribution verification (verifyAttribution):

• verifyAttribution(actingUserHomeInstance, sourceInstance) normalizes both via extractDomain and compares
• Two valid cases:
  1. Direct: authorDomain === sourceDomain — standard S2S, peer sends events for its own users
  2. Homeward relay: authorDomain === extractDomain(getOurOrigin()) — a client-federation user sent a message on a remote server, and the S2S relay forwards it back to the author's home instance
• Applied as the FIRST check in every relay event processor (13+ handlers) — before user resolution or DB writes
• Prevents malicious peers from forging events attributed to users on unrelated instances (neither source nor receiver)

All origin comparisons use extractDomain() or getOurOrigin() with normalization, handling both bare domains and full URLs consistently.

• federation.ts:2447 -- same pattern in processFriendRemoveEvent

---

4. DM Message Relay

1-on-1 DMs

Outbound (origin instance):

1. Message created via REST (POST /api/dm/:id/messages) or WS (dm_message_create)
2. queueDmRelay(message, channelId, 'create') called from dm.ts / events.ts
3. buildRelayPayload() constructs the message portion with homeUserId, homeInstance, content, replyToId, editedAt, createdAt
4. getDmParticipants(channelId) resolves all members to (homeUserId, homeInstance) pairs with profile snapshots
5. getGroupDmTargetOrigins(channelId) returns undefined (no owner -> broadcast to all)
6. queueOutboxEvent(messageId, channelId, 'create', payload, undefined) -> queued to ALL active peers

Channel creation (1-on-1 with federated user):

• When POST /api/dm creates a channel where either participant has homeInstance set, the deterministic federatedId = SHA256(sorted([homeUserIdA, homeUserIdB])).slice(0, 32) is computed and stored immediately
• This ensures findOrCreateDmChannel on the receiving instance finds the existing channel when the S2S reply arrives, preventing duplicate channels

Inbound (receiving instance -- processCreateEvent):

1. Validate: event.message and event.participants (>= 2) required
2. Dedup: check (sourceInstance, sourceMessageId) -- reject if exists
3. Resolve ALL participants via resolveOrCreateReplicatedUser, hydrate profiles
4. No event.federatedId -> 1-on-1 path
5. Compute deterministic federatedId = SHA256(sorted([homeUserIdA, homeUserIdB])).slice(0, 32)
6. findOrCreateDmChannel(federatedId, [localUserA.id, localUserB.id], db):
   • Find by federatedId in dm_channels
   • If exists: ensure both users are members (idempotent insert)
   • If not: create channel with federatedId, add both members
7. Insert dm_messages with sourceInstance and sourceMessageId
8. Process attachments (see File Replication)
9. Broadcast dm_message_created to local members, skipping members whose homeInstance === sourceInstance (they already have the original)

Group DMs

Outbound (origin instance): Same as 1-on-1 except:

• getGroupDmTargetOrigins(channelId) returns a list of peer origins that have at least one participant
• Normalizes homeInstance to full URL before comparison
• queueOutboxEvent receives targetPeerOrigins and only queues to those peers
• Payload includes federatedId (random UUID assigned at channel creation)

Inbound (receiving instance -- processCreateEvent):

1. event.federatedId present -> group DM path
2. Find channel by federatedId -- must already exist (bootstrapped by prior member_add)
3. If not found -> reject with channel_not_found
4. Insert message, broadcast to local members

Federated ID Generation (federationOutbox.ts:computeFederatedId)

// 1-on-1: deterministic 32-char hex hash
const sorted = [homeUserIdA, homeUserIdB].sort();
return sha256(sorted.join(':')).slice(0, 32);

// Group: random 36-char UUID with dashes
return crypto.randomUUID();

The format difference (32-char hash vs 36-char UUID) is used by the self-healing migration to detect channel type independently of owner_id.

Message Deduplication

Every relayed message is stored with:

• source_instance: the relay request's sourceInstance header value
• source_message_id: the event.messageId (original message ID on source instance)

The (source_instance, source_message_id) pair is checked before insertion. Duplicates are rejected with reason 'duplicate'. A unique partial index enforces this at the DB level: idx_dm_messages_source_unique ON dm_messages(source_instance, source_message_id) WHERE source_instance IS NOT NULL.

Optional message.type field (added 2026-04-29)

FederationRelayEvent.message.type?: 'user' | 'system' is optional. When present and set to 'system', processCreateEvent writes the inserted dm_messages.type column accordingly; when absent, the receiving instance defaults to 'user'.

This is a forward- and backward-compatible addition because the inbound relay endpoint (/api/federation/relay) validates only structural fields (version, events array shape, sourceInstance); unknown fields are passed through. Old peers that don't emit type produce relay events that get inserted as user messages on receiving peers (the existing default), and old peers receiving relay events from new peers ignore the field entirely. No protocol-version bump is required.

This permissiveness is intentional — the relay envelope is designed for additive evolution. Future optional fields should follow this same pattern (no schema bump, document the field here, defaults preserve old-peer behavior).

Typing Indicator Relay

Event types: dm_typing_start, dm_typing_stop

Model: Fire-and-forget, same as call signaling. No outbox, no retry, no mutation log. Typing is ephemeral — lost packets are acceptable.

Channel identification: Uses federatedId (not instance-local dmChannelId) for cross-instance channel lookup, plus participants for resolution context.

Outbound (events.ts / dm.ts):

• handleDmTypingStart() → after local broadcast, calls sendTypingRelay(dmChannelId, 'dm_typing_start', userId)
• broadcastDmMessage() → after local dm_typing_stop broadcast, calls sendTypingRelay(dmChannelId, 'dm_typing_stop', message.userId)

sendTypingRelay() (federationOutbox.ts):

• Fetches channel's federatedId and getDmParticipants() for target resolution
• Builds FederationRelayEvent with typing: { homeUserId, homeInstance, username }
• Calls sendCallRelay(origin, [event], { peeringTimeoutMs: 0 }) for each remote peer origin — non-active peers are skipped and a background ensurePeered warm-up is kicked off instead

Inbound (federation.ts):

• processDmTypingStartEvent → look up channel by federatedId, resolve user via resolveLocalUser() (no stub creation for ephemeral events), broadcast dm_typing to local members
• processDmTypingStopEvent → same, broadcast dm_typing_stop to local members
• Implicit clear: processCreateEvent() also emits dm_typing_stop for the message author after processing an inbound relay — primary typing clear mechanism for relayed messages

---

5. Outbox & Relay Pipeline

Event Queuing (federationOutbox.ts:queueOutboxEvent)

Trigger (API/WS handler)
  -> isFederationRelayEnabled()? No -> return silently
  -> Fetch active peers from federation_peers
  -> Filter to targetPeerOrigins (if specified) -- EXACT string match against peer.origin
  -> If zero peers match -> logs warning and returns
  -> For each peer, in a transaction:
      -> Check for existing outbox entry by (peerId, entityId)
      -> COALESCE:
          - delete + existing create -> delete both (net: never relayed)
          - update + existing create -> update payload, keep 'create' eventType
          - update + existing update -> update payload and eventType
          - no existing -> insert new entry
      -> TTL: now + (relayTtlDays * 86400000)

Coalescing Rules (per-peer, per-entity)

Incoming	Existing	Result
delete	create	Entry removed (message was never relayed)
update	create	Payload updated, keeps create type (peer gets full message)
update	update	Payload updated, type becomes latest
delete	update	Payload updated, type becomes delete
any	none	New entry inserted

Outbox Delivery Worker (federationWorker.ts:processOutboxTick)

Interval: 10 seconds (OUTBOX_INTERVAL_MS) Batch size: 50 (OUTBOX_BATCH_LIMIT) Timeout: 30 seconds per request (OUTBOX_FETCH_TIMEOUT_MS)

1. Query entries where nextRetryAt <= now joined with active peers, ordered by createdAt ASC, limit 50
2. Group by peer
3. For each peer, reconstruct FederationRelayEvent[] from stored payloads:
   • Parse JSON payload
   • Copy fields: federatedId, participants, message, reactions, reaction, membership, ownership, group, friendship, file_rejected fields
   • Set eventType, contextType, messageId, dmChannelId, encryptionVersion, timestamp
4. Build FederationRelayRequest with version: 1, sourceInstance: ourOrigin
5. Sign with buildFederationHeaders(body, peerHmacSecret, ourOrigin)
6. POST to {peerOrigin}/api/federation/relay
7. On success (200):
   • Compute the terminal entity set = accepted entries ∪ duplicate-rejected entries
   • Delete all terminal entries from outbox (matched by entityId -> outboxId)
   • Log remaining (non-duplicate) rejected entries at console.warn (they stay in outbox for retry)
   • Store result.maxUploadSize on peer record
   • Update peer: lastSeenAt = now, consecutiveFailures = 0
8. On failure (non-200 or network error):
   • handleOutboxDeliveryFailure():
     • Increment attempts per entry, compute nextRetryAt = now + backoff
     • Increment peer consecutiveFailures, set lastFailureAt
     • If consecutiveFailures >= PEER_UNREACHABLE_THRESHOLD (10) -> mark peer unreachable

Terminal rejection reasons

processOutboxTick recognizes a configurable set of receiver-acknowledged terminal rejection reasons (constant TERMINAL_REJECTION_REASONS in federationWorker.ts): duplicate, recipient_not_found, attribution_mismatch, unknown_event_type, self_target_invalid. Outbox entries with these reasons are deleted with no retry.

• duplicate — the receiving instance already has the row (same (sourceInstance, sourceMessageId)); retrying will fail identically until TTL.
• recipient_not_found, attribution_mismatch, unknown_event_type — structural mismatches that cannot be resolved by retrying.
• self_target_invalid — emitted by processFriendRequestCreateEvent when an inbound friend_request_create's from-identity equals its to-identity (after origin normalization). Defense-in-depth: the sender's local cannot_friend_self check should catch this, but the receiver does not trust upstream validation. Retrying will not change the payload. The friend-create rollback callback maps this to client-facing peer_rejected.

For non-duplicate terminals, the worker invokes a registered permanent-failure callback via invokePermanentFailureCallback(eventType, messageId, reason) from utils/federationRollback.ts. Currently registered: friend_request_create → rollbackFriendRequestCreate (deletes the local friend_requests row by relay_message_id and emits WS friend_request_relay_failed to the sender). Other event types may register their own callbacks. See social.md §6 "Failure Handling" for the friend-specific rollback contract.

Ghost-row failure mode. Rollback callbacks are best-effort: the registry catches and logs callback errors but does NOT re-throw. A botched rollback (e.g., DB write fails mid-rollback due to disk full or FK violation) can leave a ghost row that no longer corresponds to any in-flight relay. Acceptable vs. retry-forever blocking the outbox, but worth knowing when debugging stuck pending states.

5xx and network failures are NOT terminal — those use the existing exponential-backoff retry path. Pending operations stay pending indefinitely under sustained connectivity loss, matching the pre-existing DM relay behavior.

Logged at console.log ("outbox entry removed (terminal)") to distinguish from retained-for-retry console.warn messages.

Retry Backoff Schedule

Attempt	Delay
1	30 seconds
2	1 minute
3	5 minutes
4	15 minutes
5	1 hour
6	6 hours
7+	24 hours (cap)

The schedule above (BACKOFF_SCHEDULE_MS) paces per-entry retries. Peer-level recovery from unreachable is separate and demand-driven: processRecoveryTick probes unreachable peers on RECOVERY_BACKOFF_MS = [30s, 1m, 5m, 15m] while they have queued mail (15-min backstop when silent), paced by the per-peer last_probe_at / probe_attempts columns and the probePeerReachable helper in utils/federationRecovery.ts. See PEER_UNREACHABLE_THRESHOLD.

Authentication-failure handling (401 / 403)

When a relay response is 401 (HMAC rejected) or 403 (remote's peer row is non-active or missing), the worker increments consecutive_auth_failures on the peer row, applies backoff to the queued outbox entries via the existing BACKOFF_SCHEDULE_MS, and preserves hmac_secret. After AUTH_FAILURE_THRESHOLD = 5 consecutive auth failures (~21.5 min with the existing schedule), the peer transitions to needs_attention:

• Outbox delivery halts (the existing status = 'active' filter on the delivery query excludes needs_attention).
• hmac_secret is preserved (admin can inspect; no silent rotation).
• Affected local users receive federation_peer_rejected WS events with reason "Federation trust broken — admin must reset peering".
• Admins receive federation_peers_changed.

The worker NEVER re-handshakes via unauthenticated /peer/accept in response to a 401/403. The safeguard at /peer/accept (idempotent-200-no-update on active OR needs_attention peers) is what prevents silent HMAC rotation; the worker's job is to respect that signal and surface it to admins rather than loop. Recovery is via the admin "Reset peering" action, which deletes the local peer row and requires out-of-band re-peering.

Network failures (timeouts, non-401/403 non-2xx responses) are tracked separately via consecutive_failures and lead to unreachable at PEER_UNREACHABLE_THRESHOLD = 10. A successful delivery resets both counters.

Relay Request/Response Format

Request:

interface FederationRelayRequest {
  version: 1;
  sourceInstance: string;          // Full URL, e.g., "https://nova.ddns.net"
  events: FederationRelayEvent[];  // Max 50 per batch
}

Response:

interface FederationRelayResponse {
  accepted: string[];              // messageIds successfully processed
  rejected: Array<{
    messageId: string;
    reason: string;                // e.g., 'duplicate', 'unknown_message', 'missing_participants'
  }>;
  undeliverable?: Array<{          // optional — omitted when empty; call-signaling only
    messageId: string;
    reason: string;                // e.g., 'no_recipient'
  }>;
  maxUploadSize: number;           // This instance's max upload size in bytes
}

undeliverable bucket (call-signaling only)

In addition to accepted and rejected, the relay response may include an optional undeliverable: Array<{messageId, reason}>. Three-way classification, non-overlapping: each messageId appears in exactly one of the three arrays.

Bucket	Meaning	Retry?
accepted	Processed cleanly, ≥1 recipient reached.	No
rejected	Refused at data/protocol layer (schema, attribution, channel-not-found, etc.).	Terminal.
undeliverable	Processed cleanly, zero recipients reachable.	No — call-signaling specific.

Currently used only for dm_call_start:

• Path A (local DM exists): if no local non-caller member has an active WS connection, the event is pushed to undeliverable with reason no_recipient instead of being silently accepted. No FederatedCallEntry is created.
• Path B (no local DM): the zero-participant-match early return pushes to undeliverable rather than accepted.

Other event types (messages, reactions, friend events, profile updates, etc.) keep existing semantics — a message to an offline user is still accepted, since messages persist and re-deliver on reconnect.

The field is optional on the wire. Old peers omit it; new peers include it only when non-empty. Caller-side sendCallRelay parses the field (defaulting to an empty array when missing), so upgrade skew is a no-op until both sides are on new code.

Inbound Relay Dispatch (POST /api/federation/relay)

Body limit: 10 MB. Max 50 events per batch. Rate-limited to 90 requests/min per peer (sliding window, keyed by peer.origin). Returns 429 when exceeded. Raised from 30 after FED-009 reduced the outbox worker interval from 10s to 1s — a busy sender can now hit 60 req/min during sustained traffic.

eventType	Processor	contextType
create	processCreateEvent	dm
update	processUpdateEvent	dm
delete	processDeleteEvent	dm
reaction_add	processReactionAddEvent	dm
reaction_remove	processReactionRemoveEvent	dm
member_add	processMemberAddEvent	dm
member_remove	processMemberRemoveEvent	dm
ownership_transfer	processOwnershipTransferEvent	dm
read_state_update	processReadStateUpdateEvent	dm
friend_request_create	processFriendRequestCreateEvent	friend
friend_request_update	processFriendRequestUpdateEvent	friend
friend_request_cancel	processFriendRequestCancelEvent	friend
friend_add	processFriendAddEvent	friend
friend_remove	processFriendRemoveEvent	friend
file_rejected	processFileRejectedEvent	dm
dm_typing_start	processDmTypingStartEvent	dm (fire-and-forget, no outbox)
dm_typing_stop	processDmTypingStopEvent	dm (fire-and-forget, no outbox)
dm_close	processDmCloseEvent	dm
dm_reopen	processDmReopenEvent	dm
group_metadata_update	processGroupMetadataUpdateEvent	dm

After processing all events, the relay endpoint updates the peer's lastSeenAt and resets consecutiveFailures, then returns accepted/rejected arrays plus maxUploadSize.

---

6. Group DM Lifecycle over Federation

member_add (processMemberAddEvent -- federation.ts:1618)

Required fields: event.federatedId, event.membership.user

Two paths:

Bootstrap path (channel does not exist locally by federatedId):

1. Requires event.group metadata (owner + full member roster + group metadata snapshot)
2. Creates dm_channels row with federatedId, ownerId (resolved via resolveOrCreateReplicatedUser), ownerHomeUserId, ownerHomeInstance, plus the bootstrap name, icon, and metadataUpdatedAt from event.group
3. When event.group.icon is non-null, mirrors processGroupMetadataUpdateEvent and calls downloadProfileAsset(icon, sourceInstance) — stores the local bare filename on success or the absolute URL on failure
4. Adds ALL roster members from event.group.members (each resolved via resolveOrCreateReplicatedUser)
5. Sends dm_channel_created to local-only members (home instance matches getOurOrigin(), with normalization for bare domain)
6. Sets bootstrapped = true to skip redundant system messages and member_add broadcasts below

FederationGroupPayload (extended)

The event.group payload that bootstraps a brand-new replica carries the current metadata snapshot, so a peer that has never seen the channel materializes it with the right name and icon — no separate group_metadata_update round-trip needed:

interface FederationGroupPayload {
  owner: FederationRelayParticipant;
  members: FederationRelayParticipant[];
  // Extended for the polish pass:
  name: string | null;            // explicit null = no custom name
  icon: string | null;            // absolute URL on the wire; null = no custom icon
  metadataUpdatedAt: number;      // 0 for legacy/unset rows
}

Older peers that omit these fields fall back to safe defaults (null name/icon, metadataUpdatedAt = 0). Receivers never re-relay these fields — only the owner's home instance authors group_metadata_update events.

Incremental path (channel already exists):

1. Validates authority: any HMAC-verified peer is accepted (relaxed — the sourceInstance === channel.ownerHomeInstance check was removed to support cross-instance access). The per-user attribution check (verifyAttribution) still applies.
2. Cancels soft-delete if channel was pending GC
3. Resolves added user via resolveOrCreateReplicatedUser
4. Enforces max 10 members
5. Inserts dm_members row (idempotent -- skip if exists)
6. Inserts system message, broadcasts dm_member_added to local WebSocket clients

member_remove (processMemberRemoveEvent -- federation.ts:1825)

1. Find channel by federatedId -- if not found, accept idempotently
2. Validate authority: owner's instance for kicks (reason !== 'leave'), any instance for self-leave
3. Resolve user via resolveLocalUser -- if not found, accept idempotently
4. Insert system message (before deletion, so broadcast includes the leaving user)
5. Delete dm_members row, clean up read_states
6. Broadcast dm_member_removed to remaining local members
7. If zero members remain -> soft-delete channel (deletedAt = now)

ownership_transfer (processOwnershipTransferEvent -- federation.ts:1938)

1. Find channel by federatedId -- if not found, accept idempotently
2. Validate authority: normalizeOriginForCompare(sourceInstance) === normalizeOriginForCompare(channel.ownerHomeInstance). Both sides are normalized to handle the bare-vs-full storage convention (see dm-system.md historical bugs for why this matters).
3. Resolve new owner via resolveOrCreateReplicatedUser (never resolveLocalUser -- must guarantee valid ID)
4. Update dm_channels: ownerId, ownerHomeUserId, ownerHomeInstance (canonicalized to full URL on storage via canonicalizeHomeInstance so future authority checks stay stable)
5. Broadcast dm_owner_updated WebSocket event with newOwnerHomeUserId + newOwnerHomeInstance so local clients can refresh their owner-routing cache without reconnecting
6. Insert system message with previous owner as actor

Triggered by both auto-transfer-on-leave and the manual POST /api/dm/:id/transfer endpoint — the receiver path is the same.

group_metadata_update (processGroupMetadataUpdateEvent)

Owner-authored update of a group DM's name and/or icon. Mirrors the profile_update shape — every payload carries both fields (null is unambiguously "cleared"), and a server-side version vector handles dedup; there is no partial-update wire form.

Payload: FederationGroupMetadataPayload:

• name: string | null — trimmed, length-checked at the owner instance and re-checked by the receiver
• icon: string | null — absolute http(s) URL on the wire (the owner instance normalizes its bare-filename storage via normalizeIconForWire); receivers download and store a bare filename, or fall back to the absolute URL on download failure
• metadataUpdatedAt: number — captured at the moment of the owner-instance DB write
• actor: FederationRelayParticipant — owner by authority invariant; used only for system-message rendering

Authority: extractDomain(sourceInstance) === extractDomain(channel.ownerHomeInstance). Otherwise rejected as attribution_mismatch. Receivers never re-relay this event — the owner instance is the only emitter.

Targeting: getGroupDmTargetOrigins(channelId) — every peer that hosts a member of the channel.

Coalescing: queued via queueGroupMetadataRelay; rapid edits coalesce per peer with entityId = channelId.

Receiver flow (processGroupMetadataUpdateEvent):

1. Lookup channel by event.federatedId. If missing → accepted (idempotent — no replica to update).
2. Authority: domain match against channel.ownerHomeInstance. Mismatch → rejected as attribution_mismatch.
3. Receiver hardening (don't trust remote peers):
   • missing event.metadata → rejected as missing_metadata_payload
   • payload.name non-null with trimmed length outside [GROUP_DM_NAME_MIN_LENGTH, GROUP_DM_NAME_MAX_LENGTH] → rejected as invalid_payload
   • payload.icon non-null without http:// or https:// prefix → rejected as invalid_payload
4. Version check: payload.metadataUpdatedAt > channel.metadataUpdatedAt. Stale or duplicate → accepted silently (no side effects).
5. Diff against the stored row. If neither name nor icon actually changed → accepted; no system message; no broadcast.
6. Idempotency dedup pre-check on (sourceInstance, sourceMessageId) for both suffixes (see below). If every changed field already has its corresponding system row → accepted silently (outbox retry / initial-sync replay path).
7. Resolve icon: when iconChanged and payload.icon !== null, downloadProfileAsset(payload.icon, sourceInstance). Local filename on success; absolute URL fallback on failure (mirrors processProfileUpdateEvent).
8. Resolve actor → local user id via resolveOrCreateReplicatedUser. On failure (e.g. tombstoned identity), fall back to channel.ownerId. If both are null → rejected as actor_not_found.
9. Single transaction: update dm_channels.{name, icon, metadataUpdatedAt}; insert one or two system messages tagged with (sourceInstance, sourceMessageId) using the suffix scheme ${event.messageId}:name / ${event.messageId}:icon so two changes in a single event yield two distinct dedup keys.
10. Broadcast dm_channel_updated to local members via sendToDmMembers.
11. Broadcast each new system message via dm_message_created.
12. Cleanup old local icon file when the icon changed away from a bare filename (matches the avatar precedent at users.ts:463-466).

Wire dump (illustrative):

{
  "messageId": "1840000000000000123",
  "eventType": "group_metadata_update",
  "federatedId": "9b8c2f4e-1a2b-4c3d-9e8f-0a1b2c3d4e5f",
  "metadata": {
    "name": "weekend plans",
    "icon": "https://nova.ddns.net/api/uploads/abc123.png",
    "metadataUpdatedAt": 1746864000000,
    "actor": {
      "userId": "...",
      "homeUserId": "...",
      "homeInstance": "https://nova.ddns.net",
      "profile": { "username": "heidi" }
    }
  }
}

Local-Only Broadcast Principle

Users connected to multiple instances must see each DM channel exactly once (from their home instance). All structural broadcasts (dm_channel_created, system messages) filter to local members only:

const isLocalMember = (u: { homeInstance?: string | null }) =>
  !u.homeInstance || !domainOrigin ||
  u.homeInstance === domainOrigin ||
  `https://${u.homeInstance}` === domainOrigin;

Does NOT apply to: Regular DM messages (dm_message_created for user messages). These broadcast to all local dm_members regardless of home instance.

System Messages

System messages (type = 'system' in dm_messages) are instance-local -- they are NOT relayed via federation. Each instance creates its own when processing events.

Event	Content JSON	Actor (userId)
member_added	{event, targetUserId, targetDisplayName}	User who added them
member_removed	{event, targetUserId, targetDisplayName, reason}	User who left/was removed
owner_changed	{event, newOwnerId, newOwnerDisplayName}	Previous owner
name_changed	{event, oldName, newName}	Owner who renamed
icon_changed	{event}	Owner who set/cleared the icon

The name_changed and icon_changed rows are created by both PATCH /api/dm/:id (origin instance) and processGroupMetadataUpdateEvent (receivers). On receivers they are dedup-tagged with (sourceInstance, ${event.messageId}:name) and (sourceInstance, ${event.messageId}:icon) so retries and initial-sync replay don't double-insert.

Outbound Queuing (Origin Instance -- dm.ts)

When a group DM is created or modified locally, the origin instance queues federation events:

Group DM creation (POST /api/dm/group):

• Iterates each remote target user (those with homeInstance !== domainOrigin)
• Builds a member_add event per remote user, carrying the full roster in event.group
• Computes finalTargets by starting from getGroupDmTargetOrigins() and adding the new member's normalized homeInstance
• Calls appendMutationLog + queueOutboxEvent per event

Add member to existing group (POST /api/dm/:id/members):

• Same structure as creation -- builds member_add with full group metadata
• Normalizes new member's homeInstance to full URL before including in targets

Leave group (DELETE /api/dm/:id/members):

• Computes fedTargetOrigins before deleting the member (so the leaving user's peer is still included)
• Queues member_remove event with reason: 'leave'

Ownership transfer (auto-on-leave; manual via POST /api/dm/:id/transfer):

• Queues ownership_transfer event with previousOwner and newOwner

Group metadata update (PATCH /api/dm/:id):

• Owner-only. Queues group_metadata_update via queueGroupMetadataRelay(channelId, { name, icon, metadataUpdatedAt, actor }) after the local DB write
• targetOrigins = getGroupDmTargetOrigins(channelId) — every peer hosting a member
• Receivers re-validate name length / icon URL scheme on inbound (see processGroupMetadataUpdateEvent above)

---

7. File Replication

Outbound (origin instance)

When queueDmRelay constructs the relay payload, each attachment gets a sourceUrl:

sourceUrl: `${getOurOrigin()}/api/uploads/${attachment.filename}`

The payload also carries playable (video web-playability, computed by the origin from the probed codec — see uploads.md §3) so the receiving instance need not re-probe the file's codec.

Inbound (receiving instance -- processCreateEvent)

1. For each attachment in event.message.attachments:
   • SSRF check: isUrlFromPeer(sourceUrl, peerOrigin) -- hostname of sourceUrl must match peer origin hostname
   • Create attachments row with filename = sourceUrl (remote URL as interim filename), carrying through playable from the relay payload
   • Queue federation_file_queue entry with status = 'pending', expiresAt = now + 30 days
2. Initial WebSocket broadcast uses sourceUrl directly (frontend's AttachmentRenderer detects http prefix)

File Download Worker (federationWorker.ts:processFileQueueEntry)

Interval: 30 seconds. Batch: 5 files. Timeout: 60 seconds per download.

1. SSRF protection: validate sourceUrl hostname matches peerOrigin hostname
2. Pre-download size check against maxUploadSizeBytes from instance settings
3. Download via fetch with streaming pipeline to disk (Readable.fromWeb -> fs.createWriteStream)
4. Post-download size verification (defense in depth)
5. Generate thumbnail via sharp (same as local upload flow)
6. Update attachments row: filename = localFilename, size, thumbnailFilename
7. Fallback: if no existing attachment row was found (legacy queue entry), insert a new one
8. Mark file queue entry as completed with targetFilename
9. Broadcast dm_message_updated to refresh client-side attachment display

Size Rejection Flow (handleSizeRejection)

When a file exceeds the local instance's size limit:

1. Mark file queue entry as rejected with reason = 'size_limit_exceeded'
2. Update local attachment: federationStatus = 'remote', federationMeta = source info JSON
3. Determine affected local users (native to this instance -- !user.homeInstance || user.homeInstance === ourOrigin)
4. Queue file_rejected reverse relay event to the sender's instance (sourceInstance)
5. Broadcast dm_message_updated locally so clients see the 'remote' badge

Inbound file_rejected (processFileRejectedEvent -- federation.ts:2458)

When the origin instance receives a file_rejected event:

1. Find local message by event.messageId (the original local message ID)
2. Match attachment by sourceFilename or fallback to single attachment
3. Resolve affectedUserIds (homeUserIds) to local replicated user stubs
4. Merge rejection info into federationMeta (accumulates from multiple peers)
5. Set federationStatus = 'remote_partial'
6. Broadcast dm_message_updated + targeted federation_file_rejected toast to message author

Federation Status on Attachments

Status	Meaning
null	Local upload, no federation involvement
'local'	Successfully downloaded from peer
'remote'	Rejected (size limit), federationMeta has source instance info
'remote_partial'	Rejected by some peers, federationMeta has per-user rejection array

File Download Retry

Uses the same backoff schedule as outbox delivery. Max attempts: 10 (MAX_FILE_ATTEMPTS). After exceeding max attempts: status = 'failed', rejectionReason = 'max_attempts_exceeded'.

Boundary vs. tus Upload Migration (2026-04)

The federation_file_queue worker downloads files from peer instances via plain fetch + Readable.fromWeb to disk, post-completion. It consumes finished files at ${uploadDir}/${filename}. Whether a file got there via the legacy multipart endpoint, the current tus endpoint at /api/files/*, or any future protocol is invisible to this worker -- its contract is purely with the on-disk filename. No changes were required when the upload protocol changed.

---

8. Read State Relay

read_state_update Event

When a user marks a DM channel as read (channel_ack) or marks it unread (mark_unread), the read state is relayed to all peer instances so cross-instance sessions stay in sync.

Outbound (events.ts:handleChannelAck / handleMarkUnread):

• Fires after writing read_states locally
• Only triggers for DM channels with a federatedId (cross-instance DMs)
• Calls queueReadStateRelay(channelId, messageId, userId) in federationOutbox.ts
• Queued via the standard outbox pipeline — durable, retried by the background worker
• Entity key read_state:{federatedId}:{userId} enables coalescing (rapid acks collapse to latest)
• mark_unread with the '0' sentinel (delete read state entirely) is NOT relayed — it cannot be mapped to a message

Event payload:

{
  eventType: 'read_state_update',
  dmChannelId: string,
  messageId: string,            // unique event ID: 'read_state:{userId}:{timestamp}'
  federatedId: string,          // DM channel's federatedId (cross-instance channel lookup)
  encryptionVersion: 0,
  timestamp: number,            // LWW tiebreaker
  readState: {
    user: { homeUserId: string; homeInstance: string },
    messageRef: { sourceInstance: string; sourceMessageId: string }
  }
}

messageRef identifies the acked message in federation coordinates. If the message originated on this instance, sourceInstance is our own origin and sourceMessageId is the local message ID. If the message was relayed here, sourceInstance and sourceMessageId come from the dm_messages row's source_instance/source_message_id columns.

Inbound (processReadStateUpdateEvent):

1. Resolve channel by federatedId — reject if not found
2. Resolve user via resolveLocalUser — reject if not found
3. Translate messageRef to local message ID:
   • If sourceInstance matches our origin: sourceMessageId IS our local ID
   • Otherwise: look up dm_messages by source_instance + source_message_id
4. If no local message found (relay hasn't arrived yet): silently accept (no-op)
5. Upsert read_states using timestamp-only LWW (event.timestamp > existing.updatedAt)
6. Echo channel_ack to the user's local WebSocket connections (multi-tab sync)

---

8b. DM Close/Reopen Relay

Overview

When a user closes or reopens a DM on their home instance, the action is relayed to all peer instances that hold a copy of the channel. This keeps the visibility state of a DM consistent across all instances that participate in it.

Only DMs with a federatedId are eligible. Legacy local-only DMs (created before federation was added, with no federatedId) are silently skipped.

Event Types

Event	Trigger
dm_close	User calls DELETE /api/dm/:id (soft-close)
dm_reopen	User calls POST /api/dm and reopens a closed 1-on-1 DM

Payload

{
  eventType: 'dm_close' | 'dm_reopen',
  dmChannelId: string,          // local channel ID (context only)
  federatedId: string,          // cross-instance channel lookup key
  messageId: string,            // unique event ID: 'dm_close:{federatedId}:{userId}:{ts}'
  encryptionVersion: 0,
  timestamp: number,
  dmCloseReopen: {
    homeUserId: string,         // acting user's home user ID
    homeInstance: string,       // acting user's home instance (full URL)
  }
}

Outbound (federationOutbox.ts:queueDmCloseRelay)

Called from dm.ts after the local close or reopen is committed.

1. Fetch the channel's federatedId — if null (local-only DM), return silently
2. Fetch the acting user's (homeUserId, homeInstance) federation identity
3. Build FederationRelayEvent with eventType and dmCloseReopen payload
4. getGroupDmTargetOrigins(dmChannelId) resolves the delivery targets:
   • 1-on-1 DMs (ownerId = NULL): returns undefined → broadcast to ALL active peers
   • Group DMs: returns the set of peer origins that have at least one participant
5. Enqueue via appendMutationLog + queueOutboxEvent

Inbound

processDmCloseEvent (federation.ts):

1. Look up channel by federatedId — if not found, accept silently (idempotent)
2. Resolve acting user via resolveLocalUser (lookup-only; no stub creation for close/reopen) — if not found, accept silently
3. If the user has no dm_members row in this channel, accept silently
4. Set dm_members.closed = 1 for the resolved local user
5. Broadcast dm_channel_closed to the user's local WebSocket connections

processDmReopenEvent (federation.ts):

1. Look up channel by federatedId — if not found, accept silently
2. Resolve acting user via resolveLocalUser — if not found, accept silently
3. If the user has no dm_members row, accept silently
4. Set dm_members.closed = 0
5. Build full DmChannel payload and broadcast dm_channel_created to the user's local WebSocket connections (mirrors the automatic reopen path in broadcastDmMessage)

Closed-State Reopen on Message Relay (Bug Fix)

processCreateEvent (inbound message relay) now mirrors the local broadcastDmMessage logic: before broadcasting dm_message_created, it checks each recipient's dm_members.closed flag. For any recipient with closed = 1:

1. Set closed = 0
2. Broadcast dm_channel_created (with the new message as lastMessage) to resurface the DM in the recipient's sidebar
3. Then broadcast dm_message_created

This fixes a gap where relayed messages bypassed the closed-state reopen logic, leaving the DM hidden for recipients whose closed flag was set on the receiving instance.

---

9. Friend Relay

Event Flow (social.ts)

User Action	Federation Event	Authority Check
Send friend request	friend_request_create	from.homeInstance === sourceInstance
Accept/decline request	friend_request_update	to.homeInstance === sourceInstance
Cancel outgoing request	friend_request_cancel	from.homeInstance === sourceInstance
Accept creates friendship	friend_add	to.homeInstance === sourceInstance
Remove friend	friend_remove	Either side's instance

Target Resolution (getFriendEventTargets)

Computes which peer origins need the event. Compares fromHomeInstance and toHomeInstance against getOurOrigin() with normalization applied, correctly handling both bare domain and full URL formats.

Context ID

Friend events use a deterministic context ID: friend:${sorted[homeUserIdA, homeUserIdB].join(':')}.

Outbound Payload Construction

Each friend endpoint builds a FederationRelayEvent with:

• contextType: 'friend'
• friendship payload containing from and to as FederationRelayParticipant objects
• fromProfile and/or toProfile snapshots (FederationRelayProfileSnapshot)
• entityId formatted as friend_req:${sorted_ids}:${timestamp} (for requests) or friend_remove:${sorted_ids}:${timestamp}

The full event payload is stored in both appendMutationLog (for sync) and queueOutboxEvent (for delivery).

Inbound Processing

processFriendRequestCreateEvent (federation.ts:2082):

• Authority check: from.homeInstance !== sourceInstance -> reject
• Resolve sender via resolveOrCreateReplicatedUser + hydrate profile
• Resolve recipient via resolveLocalUser (must be native to this instance)
• Idempotency: if already friends or pending request exists, accept as no-op
• Create friend_requests row, broadcast friend_request_received to recipient

processFriendRequestUpdateEvent (federation.ts:2178):

• Authority check: to.homeInstance !== sourceInstance -> reject
• Resolve sender (original requester) via resolveLocalUser (must exist locally)
• Resolve recipient (acceptor/decliner) via resolveOrCreateReplicatedUser
• Find pending request, update status
• Broadcast friend_request_accepted or friend_request_declined to the original sender

processFriendRequestCancelEvent (federation.ts:2254):

• Authority check: from.homeInstance !== sourceInstance -> reject
• Both users must exist locally. If not, accept idempotently.
• Delete the pending friend request. Broadcast friend_request_cancelled to recipient.

processFriendAddEvent (federation.ts:2318):

• Authority check: to.homeInstance !== sourceInstance -> reject
• Resolve both users via resolveOrCreateReplicatedUser + hydrate profiles
• Insert friends row (idempotent)
• Auto-resolve any pending friend_requests to 'accepted' (handles out-of-order delivery)
• Determine which user is local (from.homeInstance === ourOrigin) and broadcast friend_request_accepted

processFriendRemoveEvent (federation.ts:2404):

• Authority check: either from.homeInstance or to.homeInstance must be sourceInstance
• Both users resolved via resolveLocalUser. If not found, accept idempotently.
• Delete friends row in both directions
• Determine local user (whose homeInstance is NOT the source) and broadcast friend_removed

---

10. Profile Sync

Profile sync uses two mechanisms that operate independently:

S2S Profile Hydration (Server-side)

When relay events carry FederationRelayProfileSnapshot data:

• processCreateEvent: hydrates participant profiles on message relay
• processFriendRequestCreateEvent / processFriendAddEvent: hydrates friend profiles
• social.ts federated friend-request handler: hydrates the looked-up stub

hydrateReplicatedUserProfile is best-effort fill-empty only: it never overwrites an existing avatar/banner/displayName/bio. This protects locally-downloaded bare filenames produced by processProfileUpdateEvent (which carries the monotonic profileUpdatedAt version) from being clobbered back to absolute URLs on the next DM/friend relay. Authoritative updates flow exclusively through the version-checked profile_update event.

Detached-account guard: Alongside the native-user skip (!user.homeInstance → return), the function also no-ops on detached rows (federation_home_orphaned = 1 → return user). A detached account retains its homeInstance for provenance (design §7), so the native-user skip alone would not catch it; without this guard a DM/friend relay from the reset domain's new incarnation, resolved via an old homeUserId tier-1 hit, could fill the sovereign account's empty fields. This is the same domain-keyed mutation class as profile_update/presence_update, guarded at its own site (design §4.3).

When the function does fill an empty avatar/banner, it calls downloadProfileAsset against the user's home instance and stores the resulting local bare filename. Only on download failure does it fall back to the absolute URL — matching the behavior of processProfileUpdateEvent.

Profile Sync (S2S)

Profile data is synced server-to-server. The home instance is authoritative — when a user updates their profile, the home server queues a profile_update relay event to all active peers via the outbox.

Event: profile_update (contextType: profile)

Payload: FederationProfileUpdatePayload:

• homeUserId, homeInstance, profileUpdatedAt (monotonic version)
• username — the home user's canonical handle (without @domain). Receivers apply displayName ?? username when writing the stub's displayName, so stubs whose home user has no displayName show the real handle instead of getting clobbered to null. Username itself is immutable on the home instance, so the receiver does NOT rewrite the stub's username column on profile_update.
• displayName, avatar (absolute URL or null), banner (absolute URL or null)
• accentColor, avatarColor, bio

Targeting: Broadcast to all active peers. Peers silently accept if they have no replica.

Coalescing: entityId = homeUserId. Rapid successive edits coalesce to one delivery per peer.

Processing: Remote overwrites all 6 mutable fields unconditionally; displayName falls back to payload.displayName ?? payload.username. Rejects if incoming profileUpdatedAt ≤ stored. Broadcasts user_updated to local WS clients.

Detached-account guard: After the domain-collision check and before the version check, if the resolved localUser has federation_home_orphaned = 1 (detached — home domain was reset, now a sovereign local account), the event is acked (messageId pushed to accepted) and skipped without applying. The reset domain's new incarnation must never overwrite an established account's profile by replaying its old homeUserId. Ack rather than reject because the sender legitimately considers the identity theirs to update; from this side the update simply no-ops.

Profile Image File Replication

When a profile_update relay carries avatar or banner absolute URLs, the receiving instance downloads the image files locally rather than storing remote URLs. This eliminates cross-origin dependencies — avatars render from the local /api/uploads/ endpoint.

Flow: processProfileUpdateEvent calls downloadProfileAsset(url, sourceInstance) for each of avatar/banner:

1. SSRF check (URL hostname must match authenticated source instance)
2. fetch with 10s timeout
3. Content-type validation (image/* only)
4. Stream to temp file (temp_{snowflake}{ext}), atomic rename to {snowflake}{ext}
5. Store local bare filename in user row

Fallback: On any download failure (timeout, HTTP error, non-image content, SSRF mismatch), the absolute URL is stored instead. This degrades to the pre-replication behavior — the avatar loads cross-origin from the home instance.

File cleanup: When avatar/banner changes, the old local file is deleted via deleteUploadFile(). The check !oldValue.startsWith('http') ensures only locally-downloaded files are deleted, not absolute URL strings.

Migration / backfill: backfillReplicatedProfileAssets runs on server start (kicked off by startFederationWorkers, non-blocking). It scans users rows where home_instance is set and avatar/banner start with http, calls downloadProfileAsset against the home instance for each, and rewrites successful downloads to local filenames. Idempotent: rows whose home is unreachable are left as URLs (they still render while the peer is up) and retried on the next start. This handles both legacy data from before file replication shipped and any rows whose home was offline during a previous attempt.

Write protection: Remote instances reject PATCH /users/@me profile field updates for replicated users (homeInstance set). Profile data is read-only on remote — only updated via S2S relay.

Bootstrap: When a new origin appears in a user's replicatedInstances, the home server queues a targeted profile_update to that peer.

File handling: Profile images are downloaded locally on relay receipt (see "Profile Image File Replication" above).

Replaces: Client-driven profileSync.ts (deleted). hydrateReplicatedUserProfile still bootstraps null fields during DM/friend relay — it now also runs the same local-download path so newly-created stubs end up with bare filenames, not URLs.

Presence Sync (S2S)

Native users' status (and optional rich activities) is projected to peers via the presence_update relay event. Closes the doc/code gap previously documented in activity-presence.md — replicated stubs now have their status maintained by the home instance over S2S, not derived from absent local WS state.

Event: presence_update (contextType: profile)

Payload: FederationPresenceUpdatePayload:

• homeUserId, homeInstance
• status: 'online' | 'idle' | 'dnd' | 'offline'
• activities?: Activity[] (omitted when empty)
• ts: number — emitter clock (last-write-wins per stub if needed)

Outbox-only — never written to mutation log. Presence is ephemeral. Replaying old presence on peer activation would be wrong (stale state). The outbox queues directly without appendMutationLog. Stale entries that fail delivery beyond retry budget are dropped.

Sender call sites (all in utils/federationPresence.ts:queuePresenceRelay):

• ws/handler.ts (auth path) — online
• ws/handler.ts (finalizeDisconnect) — offline
• ws/events.ts (handlePresenceUpdate) — manual online/idle/dnd
• ws/events.ts (handleActivityUpdate) — when activities change
• routes/users.ts (showActivity-toggle clear) — cleared activities

No-op for replicated users (we don't own their presence).

Targeting: broadcast to all active peers (mirrors profile_update). Peers without a stub silently no-op. Privacy: status is already public to anyone authorized to see the user via friend/DM/space relationships, so broadcast-fanout adds no new disclosure surface.

Coalescing: entityId = userId, contextId = userId. Rapid status flaps coalesce to the latest queued event per peer.

Receiver: processPresenceUpdateEvent (routes/federation.ts). Strict attribution — payload.homeInstance domain MUST equal source peer's domain. Resolves the local stub by homeUserId, validates the stub's homeInstance matches the payload's domain, updates the stub's status, and broadcasts a WS presence_update to local users via collectProfileBroadcastTargetIds(stub.id) — friends, DM members, and space co-members.

Detached-account guard: After the domain-collision check and before the status write, if the resolved stub has federation_home_orphaned = 1 (detached — home domain was reset, now a sovereign local account), the event is acked (messageId pushed to accepted) and skipped without applying. The reset domain's new incarnation must never flip an established account's presence by replaying its old homeUserId. Ack (not reject) mirrors the profile_update guard rationale — the sender considers the identity theirs, so we no-op rather than trigger a retry loop.

Peer lifecycle hooks (utils/federationPresence.ts):

• onPeerActivated invokes snapshotPresenceForPeer(origin) — emits a presence_update only for online natives that have an S2S relationship with the peer (friend/DM with a peer-stub, or replicatedInstances opt-in for the peer origin). Snapshot work scales with relationship count, not native count.
• onPeerDeactivated invokes markPeerStubsOffline(origin) — flips every stub from that peer to offline and broadcasts a local presence_update so users see them go offline immediately.
• Flap recovery semantics: onPeerActivated re-runs on every transition into active, including the 15-minute health-check unreachable → active recovery. This is load-bearing for correctness: markPeerStubsOffline ran on the prior deactivation, presence is not in the mutation log, so a fresh snapshot is the only signal that re-establishes truth. The relationship-scoped query bounds the cost.

Stub Username Backfill

Legacy stubs (created before the realname scheme shipped) used ${homeUserId}@${domain} as the local username. The current scheme uses ${realname}@${domain} (hints.username in resolveOrCreateReplicatedUser). Backfill heals existing legacy rows.

Worker: utils/federationStubBackfill.ts:backfillStubUsernamesForPeer(peerOrigin). Enumerates legacy-shaped stubs whose home_instance matches the peer's domain, asks the peer for the canonical username via lookupRemoteUserByHomeId, rewrites the stub's username and (if displayName is null) seeds displayName from the same fallback. Idempotent and collision-safe.

Hook points:

• onPeerActivated — runs per-peer on every transition to active (catches stubs whose home was unreachable on a prior pass).
• startupBootstrapSync — one-shot pass at boot for ALL currently-active peers (not just lastSyncedAt = 0 first-time peers).

New endpoint: POST /api/federation/users/by-home-id (HMAC-authenticated, rate-limited 60/min/peer). Body: { homeUserId: string }. Response: { found: false } or { found: true, user: { homeUserId, username, profile: { displayName, avatar, avatarColor, banner, bio } } }. Native non-deleted users only.

---

11. Reaction Relay

Outbound

Reactions are queued by WS event handlers in events.ts:

• dm_reaction_add -> queueOutboxEvent(reactionId, channelId, 'reaction_add', payload, targetOrigins)
• dm_reaction_remove -> queueOutboxEvent(messageId, channelId, 'reaction_remove', payload, targetOrigins)

Payload includes userId, homeUserId, emoji, createdAt, plus messageId and messageHomeInstance for cross-instance message resolution.

The mutation log entry for reactions stores a simpler payload (no messageId/messageHomeInstance), while the outbox entry carries the full reaction payload including those fields.

Inbound

processReactionAddEvent (federation.ts:1480):

1. Resolve message via resolveLocalDmMessage(canonicalMessageId, messageHomeInstance, sourceInstance, db):
   • If messageHomeInstance === getOurOrigin() -> find by local ID (the message originated here)
   • Otherwise -> find by (messageHomeInstance || sourceInstance, canonicalMessageId) tracking -- uses messageHomeInstance when available (correct origin in 3-instance relay), falls back to sourceInstance
2. Resolve reacting user via resolveLocalUser (must already exist)
3. Dedup: check existing reaction by (dmMessageId, userId, emoji)
4. Insert dm_reactions, broadcast reaction_added to local clients

processReactionRemoveEvent (federation.ts:1561):

• Same resolution logic
• Delete matching reaction, broadcast reaction_removed if changes > 0

---

12. Initial Sync

startupBootstrapSync() (federationWorker.ts)

Triggered once at server startup (async, non-blocking). Finds peers with status = 'active' and lastSyncedAt = 0 and calls onPeerActivated(peerId, 'startup_bootstrap') for each. This preserves the original startup-sync semantics while unifying the code path with all other activation sites (see "Peer Activation Recovery" below).

Peer Activation Recovery

Every transition of federation_peers.status to active invokes onPeerActivated(peerId, reason) — one handler wired at all transition sites. Two independent invariants, both unconditional:

1. resetOutboxBackoff — sets nextRetryAt = now and attempts = 0 for every outbox entry belonging to the peer. Entries that accumulated exponential backoff before the peer went unreachable are immediately eligible again. Attempts counter is also reset so a freshly-healthy peer's next failure starts at BACKOFF_SCHEDULE_MS[0] (30s), not wherever the counter left off.
2. syncPeerMutationLog — pulls missed events from the peer's /api/federation/sync endpoint since peer.lastSyncedAt. Three passes: DM, friend, profile (in that order, each paginated). lastSyncedAt advances to Date.now() on full success; stays put on transient failure so the next activation retries the same window.

Call sites (must remain exhaustive)

File	Context	Reason
routes/federation.ts	/peer/initiate 200 activation	initiate_accepted
routes/federation.ts	/peer/accept existing-rejected override	accept_rejected_override
routes/federation.ts	/peer/accept existing-awaiting_approval	accept_awaiting_approval
routes/federation.ts	/peer/accept existing-pending	accept_pending
routes/federation.ts	/peer/accept new-peer	accept_new
routes/federation.ts	/approval-requests/:id/approve success	approval_handshake
utils/federationWorker.ts	Health check unreachable → active	health_check_recovery
utils/federationPeering.ts	ensurePeered/performHandshake 200	ensure_peered
utils/federationWorker.ts	Startup scan (status=active, lastSyncedAt=0)	startup_bootstrap

HTTP handler sites dispatch fire-and-forget (.catch(log)) so the response is not blocked by sync-pull pagination. Worker-internal sites await since the worker tick is already async.

Concurrent activations for the same peer are deduplicated via an in-flight promise map keyed by peerId.

onPeerDeactivated

Mirror of onPeerActivated for the transition out of active. Invoked wherever federation_peers.status is written to unreachable, needs_attention, rejected, or revoked. Responsibility: sweep ConnectionManager.federatedCalls for entries whose federatedCallHost matches the deactivated peer and evict them — emitting dm_call_undeliverable { phase: 'host_unreachable', terminal: true } to each entry's ringedUserIds. See docs/systems/voice.md for the client teardown contract.

Call sites (exhaustive — grep onPeerDeactivated( to audit):

• utils/federationWorker.ts handleOutboxDeliveryFailure when status flips to unreachable
• utils/federationWorker.ts auth-failure path when status flips to needs_attention
• utils/federationWorker.ts resolvePendingPeers case 'rejected'
• routes/federation.ts admin revoke endpoint
• utils/federationPeering.ts performHandshake 403 PEERING_REQUIRES_APPROVAL path

Deduplicated by peerId using a separate inFlightDeactivation map (not shared with activation) so flapping peers retain clean activate-then-deactivate ordering.

A 30s periodic sentinel in federationWorker.ts (runFederatedCallSentinelTick) is the backstop — it scans active FederatedCallEntries, compares each host's current peer status against reality, and catches transitions missed by the hook sites.

Peer-state × outbox-enqueue × recovery matrix

Status	queueOutboxEvent enqueue	Mutation log captures	Recovery on transition to active
active	Queue	Yes (for covered event types — see below)	N/A
pending	Queue	Yes	onPeerActivated
unreachable	Queue	Yes	onPeerActivated
awaiting_approval	Drop, debug-log	Yes	onPeerActivated
needs_attention	Drop, debug-log	Yes	onPeerActivated (fires when the row is re-created via admin Reset + re-peer)
rejected	Drop, debug-log	Yes	onPeerActivated
revoked	Drop, debug-log	Yes	onPeerActivated (fires when the row is re-created via hard-delete + re-initiate)

queueOutboxEvent uses an exhaustive TypeScript switch on the narrowed peer-status union — adding a new status value without handling it fails compile-time typecheck (const _exhaustive: never = status;).

Mid-call race catch: when the initial peers SELECT filters out a peer because its status is non-deliverable, but the fallback loop observes the status has since flipped to active/pending/unreachable, the code re-fetches the peer row and appends it to matchedPeers so the outer enqueue loop includes it. Silent drops would lose real-time delivery under asymmetric failure (e.g., /peer/accept 200 response lost on the wire, health-check transition firing on only one side).

Mutation log coverage

Event types covered by appendMutationLog (replayed on sync-pull):

Event type	contextType	Source
DM create / update / delete	dm	federationOutbox.queueDmRelay, dm.ts delete handler
reaction_add / reaction_remove	dm	ws/events.ts
member_add / member_remove / ownership_transfer	dm	dm.ts
dm_close / dm_reopen	dm	federationOutbox.queueDmCloseRelay
read_state_update	dm	federationOutbox.queueReadStateRelay
file_rejected	dm	federationWorker.handleSizeRejection
friend_request_* / friend_add / friend_remove	friend	social.ts
profile_update	profile	routes/users.ts (PATCH /api/users/@me)

Ephemeral events (dm_typing_*, dm_call_*) are fire-and-forget by design and are NOT captured — missed typing/call-signaling packets are acceptable and carry no durable state.

/api/federation/sync contextType filter values

Filter	Returns
(none) / omitted	DM events (including dm_close, dm_reopen, read_state_update, file_rejected)
'friend'	Friend events
'profile'	Profile update events

Poison-Pill Event Handling

syncPeerMutationLog replays incoming events one at a time. If an event's inbound processor throws (e.g., UNIQUE conflict from a malformed payload, unexpected schema drift, a processor bug), the error is caught per-event, logged via console.error with the event's eventType, messageId, timestamp, peer origin, and error message, and the loop continues with the next event. lastSyncedAt is advanced past the failed event (using the event's own timestamp), so subsequent activations do not retry the poison pill.

The final Sync-pull from <origin> replayed <N> events log line is suffixed with (<K> skipped due to errors) when K > 0, surfacing the count to operators watching logs. Individual event failures are in the same logs under Skipping poison-pill event — grep console.error / stderr to recover them.

Trade-off: this policy prioritizes forward progress of the sync pipeline over strict at-least-once delivery of every mutation. An event that fails to process is silently lost to the receiving instance unless operators manually replay it (e.g., by resetting lastSyncedAt on the peer row or by reissuing the originating mutation on the sender). The alternative — refusing to advance on any error — caused the "stuck forever" state described in the pre-fix version of this section.

Sync Endpoint (POST /api/federation/sync)

HMAC-authenticated. Returns events from the federation_mutation_log.

Request:

{ sinceTimestamp: number, dmChannelId?: string, federatedId?: string, contextType?: 'dm'|'friend'|'profile', limit?: 1-500 }

Response:

{ events: FederationRelayEvent[], hasMore: boolean, checkpoint: number }

DM sync:

• Queries all dm_channels with non-null federated_id (not soft-deleted)
• Joins federation_mutation_log with dm_messages to reconstruct events
• Only returns locally-created messages (source_instance IS NULL via the LEFT JOIN)
• Handles delete mutations separately (message rows don't exist for deletes)
• For create/update: fetches current message state from DB, builds full relay event with attachments and participants
• Membership/friend mutations store the full event payload in the mutation log, so they are returned directly

Friend sync:

• Queries federation_mutation_log WHERE context_type = 'friend'
• Returns stored payloads directly (friend events carry their complete data)

Relay Event Processing

The event processing logic is extracted into processRelayEvents() (exported from federation.ts), shared by both the HTTP relay endpoint and the initial sync worker. This avoids a DNS hairpin issue where the server would HTTP-request itself through public DNS, which fails on networks without hairpin NAT.

---

13. DM Calls over Federation

DM calls work across federated instances. The caller's instance hosts the LiveKit room. Remote clients connect directly to the caller's LiveKit server using a token passed through S2S relay — no media is routed through the federation layer.

User A's client <--WS--> Instance 1 (hosts LiveKit) <--S2S HTTP--> Instance 2 <--WS--> User B's client
                              |                                                            |
                              +------------- LiveKit (direct client connection) -----------+

S2S Event Types

Four relay event types are processed in processRelayEvents():

Event Type	Direction	Key Payload Fields
dm_call_start	Host → Peers	federatedId, livekitUrl, tokens: Record<string, string> (keyed by homeUserId), caller: { homeUserId, homeInstance, displayName }, participants
dm_call_accept	Participant → Host, then Host → All Peers	federatedId, acceptor: { homeUserId, homeInstance }
dm_call_reject	Participant → Host, then Host → All Peers	federatedId, rejector: { homeUserId, homeInstance }
dm_call_end	Any → Host (if not host), then Host → All Peers	federatedId, endedBy: { homeUserId, homeInstance }

All events carry standard relay fields: eventType, messageId, encryptionVersion: 0, timestamp. All events pass through verifyAttribution() before any DB or state mutations.

Direct Delivery (No Outbox)

sendCallRelay(targetPeerOrigin, events, opts?) (federationOutbox.ts):

• Latency-sensitive: returns CallRelayResult = { ok: true } | { ok: false; reason: CallRelayFailureReason; error: string }.
• Peering resolution:
  1. If the peer row is active or unreachable, POST directly (the health check restores unreachable peers; re-handshaking is wasteful).
  2. Otherwise race ensurePeered against opts.peeringTimeoutMs (default CALL_PEERING_TIMEOUT_MS = 3_000 ms). The background handshake is not aborted on race loss — a warn-logged catch is attached so a late-rejecting background promise does not emit unhandledRejection.
• Peer-state → reason mapping is exhaustive over the EnsurePeeredResult union (active / rejected / pending / failed) plus the external timeout branch. TypeScript never check in the switch default catches future additions. Note: the livekit_unavailable reason in DmCallUndeliverableReason is emitted separately from sendFederatedCallStart's LiveKit pre-flight in ws/events.ts, not from this switch — sendCallRelay only produces CallRelayFailureReason values (peer_rejected / peer_awaiting_approval / peer_transient_failure / post_failed).
• Non-blocking mode: peeringTimeoutMs: 0 (used by typing) skips the POST for non-active peers, kicks off ensurePeered as a background warm-up, returns peer_transient_failure silently.

sendTypingRelay(dmChannelId, eventType, userId):

• Fire-and-forget to each remote DM participant's home instance via sendCallRelay(origin, [event], { peeringTimeoutMs: 0 }). Typing is an ephemeral hint — lost packets are acceptable and there is no user-facing failure surface.

Call-start failure surfacing. sendFederatedCallStart aggregates targeted-peer results and emits dm_call_undeliverable { phase: 'start' } to the caller for failed targeted peers. See docs/systems/voice.md and docs/systems/websocket.md for the event contract.

Accept / reject / end relay discipline. Every handleDmCall{Accept,Reject,End} Path-2 branch awaits sendCallRelay and emits dm_call_undeliverable { phase, terminal, failures } to the originator on failure. Accept is pessimistic-rollback (terminal: true — local FederatedCallEntry cleared, optimistic dm_call_accepted walked back via sendToFederatedCallUsers); reject and end are optimistic (terminal: false — state already cleared, informational toast only). Path-1 fan-outs via sendFederatedCallAccept / sendFederatedCallEnd / fanOutCallEvent return CallFanoutFailure[] and the host-side caller receives dm_call_undeliverable { terminal: false } listing peers that were not reached.

Ring-timeout fan-out. ConnectionManager.createDmRoom's 60 s ringing auto-clean now invokes a registered hook (setRingTimeoutFanoutHook, registered from ws/events.ts:registerCallRelayHooks) that fans dm_call_end out to remote peers, so stranded Path-A/B ringees on other instances exit their ring state instead of lingering until their own 60 s cleanup fires.

Call Flows

Start: Host validates membership, generates LiveKit tokens for all DM members (local + remote), broadcasts dm_call_incoming to local WS clients, then sends dm_call_start S2S to each remote instance with per-user tokens.

Accept: Remote instance sends dm_call_accept S2S to host. Host transitions ringing → active, broadcasts dm_call_accepted locally, fans out dm_call_accept to all other remote instances.

Reject: Remote sends dm_call_reject to host. Host destroys room, sends dm_call_end to all peers. (For 1-on-1 DMs, reject = end.)

End: Initiating instance (host or not) routes through the host. Host destroys room, fans out dm_call_end to all remote instances.

Timeout: Both host and remote instances auto-clean stale ringing calls after 60 seconds.

LiveKit Room Naming

Room name = federatedId (the cross-instance stable UUID), never the local dmChannelId (which differs per instance).

• 1-on-1 DMs: federatedId is a deterministic SHA-256 hash of the sorted homeUserId pair
• Group DMs: federatedId is a UUID assigned at creation

LiveKit Token Generation

generateFederatedCallToken(roomName, homeUserId, displayName) generates tokens with:

• TTL: 5 minutes (short join window; local calls use 1 hour)
• Room: scoped to exact federatedId
• Identity: ${homeUserId}:${displayName}
• Permissions: full DM grants (mic, camera, screen share, subscribe, data channel)

Public LiveKit URL

The URL sent in S2S payloads is always https://${DOMAIN}/livekit (the Caddy-proxied public address). The internal LIVEKIT_URL env var (http://livekit:7880) is never sent to peers.

Instances without LiveKit configured can still receive federated calls — they pass the host's URL and token to the client, which does all the heavy lifting.

In-Memory Call Registry

When a remote instance receives dm_call_start, it creates a FederatedCallEntry in memory:

interface FederatedCallEntry {
  dmChannelId: string;          // local dmChannelId for this DM
  federatedId: string;          // cross-instance room identifier
  callerId: string;             // local userId of caller's stub
  callerHomeUserId: string;
  federatedCallHost: string;    // peer origin of the host instance
  livekitUrl: string;
  tokens: Map<string, string>;  // homeUserId → LiveKit token
  state: 'ringing' | 'active';
  startedAt: number;
}

This registry ensures tokens and livekitUrl survive browser refreshes via the activeCalls array in the ready WS payload. The server filters to the per-user token at payload assembly time.

---

14. Background Workers

All workers are started by startFederationWorkers() on server boot and stopped by stopFederationWorkers() on shutdown. Each worker uses setTimeout chains (not setInterval) with abort controllers for graceful shutdown.

Worker	Interval	Batch	Timeout	Source
Outbox delivery	10s	50	30s	processOutboxTick
File download	30s	5	60s	processFileQueueTick
Health check	15min	all unreachable	10s	processHealthCheckTick
Epoch-refresh baseline	Startup + 15min (end of health tick)	active peers w/ peer_instance_id IS NULL	10s per peer	refreshPeerEpochs (populate-if-null, self-terminating)
Janitor	1h	--	--	runFederationJanitor (sync)
Startup bootstrap sync	Once at startup	--	30s per page	startupBootstrapSync → onPeerActivated

Janitor Cleanup (storageJanitor.ts:runFederationJanitor)

Target	Condition	Retention
federation_outbox	expiresAt < now	Configurable via federationRelayTtlDays (default 30)
federation_mutation_log	mutatedAt < (now - 90 days)	90 days
federation_file_queue (completed)	createdAt < (now - 7 days)	7 days
federation_file_queue (any)	expiresAt < now	30 days (set at queue time)
dm_channels (soft-deleted)	deletedAt < (now - 24h)	24-hour grace period

DM channel hard-delete cascades: reactions, embeds, attachments (DB rows + disk files), messages, members, outbox entries, mutation log entries, file queue entries.

Worker Startup Gate

startFederationWorkers() in utils/federationWorker.ts:1211 starts: outbox-delivery tick, federated-call sentinel, file-replication ticks, health-check ticks, and the storage janitor. The whole bundle is gated by process.env.DISABLE_FEDERATION_WORKERS matching '1' or 'true' at index.ts:171-176 (envBool semantics). Tests set DISABLE_FEDERATION_WORKERS=1 to silence cross-instance background traffic during setup.

Public Origin Override

PUBLIC_ORIGIN env (read via config.publicOrigin, consumed by getOurOrigin() in utils/federationAuth.ts) overrides the federation transport URL verbatim, taking precedence over the default https://${DOMAIN}. When unset, behaviour is unchanged. Intended for reverse-proxy / dev-without-TLS deployments where the public origin must be advertised explicitly (typically http://...) and differs from the bare DOMAIN value used for federated identity. The seed-peer integration harness (seedPeer.ts) does NOT use this override — see it for why localhost-port instances cannot collapse to a single peer row. The real-handshake harness (realHandshake.ts) does the opposite: it sets PUBLIC_ORIGIN to each instance's ephemeral http://127.0.0.1:<port> so the advertised sourceOrigin matches the transport, exercising the same path as production.

Handshake sourceOrigin honors this override. resolveLocalOrigin() (routes/federation.ts) delegates to getOurOrigin(), so the origin advertised in the /peer/accept handshake body is identical to the X-Federation-Origin used for authenticated S2S requests. Using https://${DOMAIN} directly (the prior behavior) desynced the responder's peer-row key from the auth origin whenever PUBLIC_ORIGIN != https://DOMAIN, causing permanent 403 Not peered. See "Trust re-establishment contract" (§1).

Test-Only Routes

POST /api/admin/test/seed-peer directly inserts a federation_peers row, skipping the multi-step peer handshake. Strictly gated: NODE_ENV='test' AND ENABLE_TEST_ROUTES='1' together; returns 404 in any other configuration. Used exclusively by the two-instance integration harness in packages/server/test/. Validates origin (must be http(s) URL), hmacSecret (≥32 chars), and status (must be one of 'active', 'pending', 'awaiting_approval', 'rejected', 'revoked', 'needs_attention', 'unreachable', 'accepted').

Rate-limit bypass (test only)

DISABLE_RATE_LIMITS=1 bypasses all @fastify/rate-limit enforcement at boot (envBool semantics: '1' or 'true'). Used by integration test harnesses where many tests share the loopback IP and would otherwise exhaust per-IP buckets across unrelated tests. Defaults to enforced rate limits in production. The two-instance harness sets this in every spawned instance's env; tests that need to assert rate-limit behaviour (e.g. Test #15) use bootTwoInstancesWithRateLimits() which omits the env var.

---

15. Settings Cache

federationOutbox.ts caches federationRelayEnabled and federationRelayTtlDays from instance_settings for 30 seconds (CACHE_TTL_MS). This prevents repeated DB reads on every message send. The cache is invalidated by TTL only -- there is no explicit cache bust on settings change.

Relevant settings in instance_settings:

Column	Default	Purpose
federation_relay_enabled	1	Master toggle for all federation relay
federation_relay_ttl_days	30	Outbox entry TTL
max_upload_size_bytes	null (uses config.maxUploadSize)	File download size limit

---

16. Client-Side Identity Helpers (identity.ts)

The frontend needs to resolve federated identities for display purposes:

parseFederatedUsername(username) -- splits "erin@nova.ddns.net" into {baseName: "erin", domain: "nova.ddns.net"}.

isSelf(user, homeUser) -- determines if a user object is the logged-in user or their replicated stub. Uses cascading checks: same ID, known self-ID set, homeInstance + baseName match.

canonicalUserMatch(a, b) -- federation-safe check for whether two user objects represent the same person. Cascades through: same local ID, homeUserId cross-match, username + homeInstance fallback.

resolveDisplayIdentity(user, homeUser) -- returns homeUser for display if user is a replicated stub of homeUser, enabling consistent avatars and display names across instances.

Cross-instance self-ID registry: registerSelfId(id) / clearSelfIds() track all Snowflake IDs belonging to the current user across connected instances, populated from WS ready events.

---

17. Self-Healing Migrations (migrate.ts)

The migration system includes several data integrity checks that run on every server startup:

Group DM ownerId repair: Detects group DMs with UUID-format federated_id (length 36, matches ________-____-____-____-____________) but NULL owner_id. Restores the owner from the first remaining member or from owner_home_user_id/owner_home_instance. Root cause: a bug in processOwnershipTransferEvent (fixed in cd7aff0) could set ownerId = NULL via resolveLocalUser fallback.

Federated ID backfill: Finds 1-on-1 DM channels without federated_id, computes deterministic SHA-256 hash from home user IDs, and sets it. Also detects relay-created duplicate channels with the same federated_id and merges messages into the oldest channel.

Duplicate channel merge: Finds federated_id values appearing on multiple channels and merges them into the oldest, moving messages, members, and cleaning up the duplicates.

Mutation log backfill: If the federation_mutation_log table exists but is empty, populates it with create entries for all existing DM messages where source_instance IS NULL (locally-created messages).

---

Known Issues

See docs/federation-production-roadmap.md for open items (FED-001 through FED-013).

• Accept/reject/end relay failures now surfaced. All three federated call-state transitions emit dm_call_undeliverable { phase, terminal, failures } to the originator on relay failure — accept rolls back optimistic state (terminal: true), reject/end keep the optimistic clear and emit an informational toast (terminal: false). See docs/systems/voice.md "Call relay failure surface" for the full contract. The host-side ring timeout also fans dm_call_end out to peers so stranded Path-A/B ringees exit the ring. One remaining edge documented in voice.md: non-host end-relay failure leaves the host's local activeDmCall marker until manual cleanup.