Adds 'rejected', 'awaiting_approval', and 'needs_attention' to the status
union, plus the consecutiveAuthFailures / autoRotateIntervalDays /
secretRotatedAt / rotationInProgress fields the UI already reads.
Tracks HMAC-failure count separately from consecutive_failures (network
errors). Auth failures and network failures have different resolution
paths; mixing them would let a single successful retry after a network
blip mask real secret desync.
Part of backlog #19 — outbox auth-failure recovery.
Note: drizzle-kit generated the 0003 SQL with an unexpected CREATE TABLE
for peer_approval_requests because the 0002 migration was authored
manually without a corresponding 0002_snapshot.json (see 14041e9). The
generated SQL has been trimmed to the single intended ALTER TABLE. The
regenerated 0003_snapshot.json correctly reflects the full current
schema, so future migrations will diff cleanly.
Closes backlog #16. Implements sendCallRelay/sendTypingRelay auto-peering
and caller-facing dm_call_undeliverable failure surface.
See internal notes
and internal notes
for the full design + implementation plan.
sendFederatedCallStart now collects per-targeted-peer results and emits
a single dm_call_undeliverable event to the caller when any targeted
peer relay fails. Destroys the local ring room when no plausible
recipient remains (no targeted success + no connected local ringee).
LiveKit pre-flight also emits via this path with reason
'livekit_unavailable' instead of a silent console.warn, closing the
60s hang for unconfigured instances.
Guards against phantom toasts when the caller cancels mid-race by
checking getRoom() before emitting.
sendCallRelay now returns CallRelayResult with a typed reason on failure.
When the peer is not already active (or unreachable), runs a racePeering
against CALL_PEERING_TIMEOUT_MS (3s). Background handshake is not aborted
on race loss — next attempt succeeds.
sendTypingRelay passes peeringTimeoutMs:0 so typing never blocks on a
handshake; instead a warm-up ensurePeered runs in the background for any
non-active peer so the NEXT relay (message, call, or typing) benefits.
Addresses code review on b22a7bd: (1) a rejected handshake now returns
{ status: 'failed', error } instead of throwing, keeping the structured
contract; (2) the "background handshake" warn only fires when the
timeout arm wins — not when the handshake is itself the race winner by
rejection. Timing tests migrated to vi.useFakeTimers for determinism.
Regression test added for the handshake-wins-by-rejection case.
Exports `racePeering(origin, timeoutMs, ensurePeeredFn?)` that races
`ensurePeered` against a deadline. On timeout, the background handshake
continues (warming the peer for the next attempt) and a warn-logged
.catch() prevents unhandledRejection. Injectable `ensurePeeredFn` param
enables full DI in tests without mocking module internals.
processMemberAddEvent, processMemberRemoveEvent, and processOwnershipTransferEvent inserted system messages unconditionally. Outbox retries and initial-sync replays (triggered whenever an admin re-approves a peering request, which recreates the peer row with lastSyncedAt=0) duplicated the system message on every delivery. Each new snowflake ID exceeded the user's last_read_message_id, flipping the channel back to unread after every deploy.
Processors now short-circuit on a matching (source_instance, source_message_id) row, and persist those fields when inserting. processMemberAddEvent emits the tagged system message in both bootstrap and incremental paths so bootstrap replays don't fall through and insert a second one; the bootstrap's dm_channel_created broadcast carries that message as lastMessage so sidebar previews and unread anchors agree across instances.
When two instances each have a native user with the same username, the
Add Friend search card for the federated one sent its request to the
local namesake instead of the intended remote user.
Root cause: `isNative = !homeUserId` in socialStore's searchUsers and
loadFriends dedup. The server backfills native users' homeUserId to
their own id so federation tier-1 lookups succeed, so `homeUserId` is
set on natives too. Only `homeInstance` distinguishes native (null)
from replicated stubs. With the wrong check, no entry was ever "native"
and the home-origin stub of the remote user was kept over the true
native record — leaving `_instanceOrigin=''`, which caused the Send
button handler to drop the domain suffix and POST to the home API,
where "nova" resolved to a completely different local user.
Also fixes loadRequests dedup to prefer the target-native record so the
search card correctly flips to "Request Pending" after sending.
Added federation_peers_changed (no-payload signal) broadcast from every
peer state mutation, and federation_approval_request_received when a new
approval request is queued. Client subscribes via onFederationPeersChanged
callback registry. FederationPanel and PendingApprovals debounce-refetch
on any event. sendToAdmins helper broadcasts only to admin users.
autoAcceptPeering means 'don't accept peering initiated by others', not
'don't initiate peering ourselves'. Two checks were incorrectly blocking
outgoing peering when auto-accept was off:
1. ensurePeered() refused to auto-initiate — reverted. When a local user
sends a DM, the server should initiate peering. The remote's
peer/accept decides whether to accept or queue.
2. queueOutboxEvent() refused to create placeholders — reverted. The
outbox needs placeholders to queue entries. Without them, DM relay
silently fails.
When both instances have autoAcceptPeering off, the approval flow
ping-ponged indefinitely. Admin A approves → handshakes to B → B
queues (202) → A's peer becomes awaiting_approval. Admin B approves →
handshakes to A → but A's gate only matched 'pending', not
'awaiting_approval', so it re-queued instead of accepting.
Now the gate matches both 'pending' and 'awaiting_approval'. When the
second admin approves and handshakes back, the first instance recognizes
its admin already approved and accepts — completing the peering.
1. queueOutboxEvent no longer creates pending peer placeholders when
autoAcceptPeering is disabled — prevents bypassing the admin's
peering control
2. Approval endpoint checks for 202 before response.ok — when the
remote also has autoAcceptPeering off, sets peer to awaiting_approval
instead of incorrectly activating it
3. awaiting_approval status added to Federation panel UI — status label,
colors, filter options so these peers are visible and manageable
When all peers are pending (no active peers with outbox entries),
processOutboxTick returned early at line 141 before reaching
resolvePendingPeers at line 302. Pending peers were never resolved
because the only code path to resolvePendingPeers was after the
active-peer delivery loop — which never ran.
getGroupDmTargetOrigins() returned undefined for 1-on-1 DMs, which
queueOutboxEvent() treated as 'broadcast to all existing peers'. When
no peers existed, nothing was queued and no handshake was ever triggered.
Now always computes target origins from DM participants so the pending
placeholder creation path runs, enabling ensurePeered() → peer/accept
→ approval queue flow.
The client's direct WS connection to remote instances (via Connections)
delivered DM events independently of S2S peering. Added activePeerOrigins
allowlist to ready payload — all DM event handlers now silently drop
events from non-home origins without an active peer. This prevents
notifications, sounds, previews, typing indicators, calls, and channel
updates from instances where peering was revoked or never established.
ensurePeered() now checks the local autoAcceptPeering setting before
initiating new peering. When disabled, only admin-explicit peer/initiate
and approval-request approve bypass this check. Closes the bypass where
client peer/ensure or outbox worker could auto-initiate outward peering
even when the admin intended to control all peering.
- C1: Include 'unreachable' peers in queueOutboxEvent query to prevent
UNIQUE constraint violation when creating placeholders
- I1: Add 'rejected' to StatusFilter in FederationPanel so admins can
see and manage rejected peers with delete/re-initiate actions
- I2: Map ensurePeered 'failed' to 'pending' in peer/ensure response
to match spec and client expectations
Adds `auto_accept_peering` integer column (default 1/true) to the
`instance_settings` singleton table, controlling whether this instance
auto-accepts incoming peering requests. Includes generated migration
`0001_clear_earthquake.sql` applied automatically on server boot.
Remove the owner-only gate on POST /api/dm/:id/members. The S2S relay
already accepts member_add from any HMAC-verified peer, and the UI
already shows the add button to all group DM members. Only the
server-side check was blocking non-owners.
The socialStore WS-driven handlers (addFriendFromAccepted,
addIncomingRequest, removeFriendLocally, removeRequestById,
updateFriendPresence) used instance-local id:origin composite keys
for deduplication. When the client is connected to multiple instances,
both fire WS events for the same federated user with different local
IDs, bypassing the dedup and creating duplicate entries.
Switch all handlers to use homeUserId??id (canonical identity),
matching the pattern loadFriends/loadRequests already use. Also
replace the loadRequests() re-fetch in updateFriendRequest with
optimistic canonical removal to avoid racing S2S relay propagation.
Remove all manual migrations — Drizzle-kit now manages schema DDL.
Data-fix migrations have all completed on both instances.
Startup initialization (settings row, worker ID, first admin)
moves to idempotent ensureDefaults().
Friends fan-out (loadFriends/loadRequests) now waits for all remote
connections to establish before querying, fixing the empty friends list
when logged into a remote instance as a federated user.
- Add _autoConnectDone wait guard to loadFriends, loadRequests, and
loadFederatedMutuals (same pattern as discoverStore)
- Add concurrency guards to prevent thundering herd from multiple
ready events firing simultaneous fan-outs
- Fix deduplication to use canonical identity (homeUserId ?? id)
instead of id:origin, preventing duplicate entries for the same
user across instances
- Auto-connect to home instance when logged in as a federated user,
with registry entry so it appears in Connections UI
- Allow re-adding error/disconnected instances in probeInstance
When a profile_update relay arrives with avatar/banner URLs, download
the files to local storage instead of storing remote absolute URLs.
Falls back to absolute URL on any download failure. Cleans up old
local files when replaced.
During room.disconnect(), LiveKit fires ParticipantDisconnected for
each remote participant BEFORE the final Disconnected event. Because
roomRef was still set, guardedUpdate() called updateParticipants(),
which updated the voiceStore while isLiveKitConnected was still true.
SoundController played user_leave for each departing participant
alongside the disconnect sound.
Fix: set roomRef.current = null before calling destroyRoom(). This
causes guardedUpdate() to return early for all teardown events.
The disconnect function handles cleanup after destroyRoom resolves.
The identity flip during disconnect: updateParticipants resolves
homeUserId → localSnowflake when activeDmCall is set, but reverts
to raw homeUserId when activeDmCall is cleared (before LiveKit
disconnect completes). SoundController sees the snowflake "leave"
and the homeUserId "join" — two phantom events for the same person.
Previous fix only checked homeUserId OR id. Now checks BOTH via
isSelf(id) which matches against a Set of {id, homeUserId}. This
recognizes the user as "self" regardless of which identity format
the participant currently has.
Root cause: SoundController compared LiveKit participant p.userId
(which is homeUserId from the home instance) against currentUser.id
(local snowflake on the current instance). For federated users these
are different IDs, so the controller thought the user's own presence
was a stranger — playing user_join/user_leave for self.
Fix: use homeUserId || id for the self-check. This matches the
LiveKit identity format used in federated calls.
Added justDisconnected guard to the participant sound loop. When
the user hangs up, isLiveKitConnected transitions to false — but
in a separate or same subscription tick, the participants list
also empties. Without the guard, SoundController plays user_leave
for every departed participant AND the disconnect sound simultaneously.
Now: if justDisconnected is true, the entire participant loop is
skipped. Only the disconnect sound plays.
RoomEvent.Disconnected handler set participants=[] and
isLiveKitConnected=false in separate setState calls. SoundController
subscription fired between them — saw empty participants while still
"connected" → played user_leave, then saw disconnected → played
disconnect. Both sounds played simultaneously.
Batching into one setState ensures SoundController sees the final
state atomically: participants gone AND disconnected in one update.
sendToFederatedCallUsers sent dm_call_ended/rejected back to the user
who initiated the action. They already disconnected in their click
handler — the redundant event triggered disconnectFn() again, causing
connect and disconnect sounds to play simultaneously.
Added excludeUserId parameter to sendToFederatedCallUsers, used in
handleDmCallEnd and handleDmCallReject Path 2.
sendFederatedCallStart was sending `https://${domain}/livekit` as the
LiveKit URL. The LiveKit SDK requires `wss://` for WebSocket connections.
The caller (local) worked because it gets the URL from config.livekit.url
(wss://). The federated acceptor failed because it used the relay URL
(https://) — the SDK can't connect over HTTPS.
Now uses config.livekit.url directly, falling back to wss:// if unset.
Bug A: handleAccept relied on dm_call_accepted server response to set
activeDmCall. But connectFn's async AudioContext resume yields to the
event loop, dm_call_accepted arrives during the yield, finds
isLiveKitConnected=false (connectFn just reset it), and skips
setActiveDmCall. The acceptor connects to LiveKit but the UI never
shows the call. Fix: set activeDmCall and clear incomingCall
directly in the click handler.
Bug B: ready handler no longer sets activeDmCall for active calls.
On refresh/restart the client has no LiveKit connection — showing
"Connecting..." with no connection is broken. The call exists on
the server but this client session is disconnected.
Four fixes addressing the full state management problem:
1. Passive ready handler: no longer auto-connects to LiveKit on
page refresh. Prevents identity conflicts when the same user
has multiple sessions fighting for one LiveKit identity slot.
The user must re-accept to join; state is shown but not acted on.
2. SoundController sync guard: incomingCallLoading/outgoingCallLoading
refs prevent multiple playSound calls during async audio load.
If call is cancelled while sound loads, stops it immediately on
completion. Eliminates the "5 ringtones at once" bug.
3. Host dm_call_accepted broadcasts now include federatedCallId so
all clients (including remote instances) can match the event.
4. Removed all diagnostic console.log statements.
Root cause: sendToFederatedCallUsers used sendToDmMembers when dmChannelId
was set, which broadcast to ALL DM members including the caller's replicated
stub. The caller's multi-instance WS received dm_call_accepted with the
REMOTE instance's dmChannelId, causing token request for a non-existent
channel (403) and preventing the caller from connecting.
Fix 1: sendToFederatedCallUsers always uses ringedUserIds (exact recipients)
instead of sendToDmMembers (all members including caller stub).
Fix 2: dm_call_accepted handler only sets activeDmCall if the client is the
caller (wasOutgoingCall) or already connected to LiveKit. Other instances of
the same user just clear ringing without entering stuck "Connecting..." state.
callOrigin was set to event.callOrigin (the HOST instance URL), which
routed accept/reject through the multi-instance WS connection. On
mobile hotspot or when the multi-instance WS drops, the accept is
silently lost — the host never knows, the call stays ringing forever.
Now callOrigin = origin (the WS that delivered dm_call_incoming).
This is always connected. The server on that instance finds the
FederatedCallEntry and relays to the host via S2S HTTP, which is
reliable and independent of client WS state.
1. Accept/reject/end from remote instance now resolves federatedId
to local dmChannelId via DB lookup, so the host can find its
VoiceRoom when the event arrives with only a federatedCallId.
Previously silently failed with "No active call" error.
2. Batch all dm_call_incoming state updates into a single
useVoiceStore.setState() call. Prevents SoundController from
starting multiple ringtone instances (async playSound guard
race when 4 separate set() calls each triggered the subscription).
3. Always overwrite callOrigin/federatedCallId (with null if absent)
on dm_call_incoming. Prevents stale values from a previous
federated call routing local accepts to the wrong instance.
Three fixes for multi-instance call state consistency:
1. Client dm_call_accepted handler only auto-connects to LiveKit if
the user was the caller (outgoingCall was set). Other instances of
the same user just clear ringing state without connecting.
2. Server processDmCallAcceptEvent remote path skips duplicate
broadcast when FederatedCallEntry is already active (prevents
state conflicts from host fan-out arriving after local accept).
3. Ready payload handler clears stuck incomingCall when restoring
an already-active call after page refresh.
Enable federated DM calls to route accept/reject/end through the correct
WebSocket connection using callOrigin, and include federatedCallId in all
dm_call payloads for server-side FederatedCallEntry lookup.
When findOrCreateDmChannel creates or finds a local channel for a
federatedId that has an active FederatedCallEntry with null dmChannelId,
update the entry. Prevents stale null references for Path B calls.
The outbox worker's event reconstruction whitelist was missing these two
fields, causing read_state_update and dm_close/dm_reopen relay events to
arrive at remote instances with empty payloads and get rejected.
Add shared isSelfOrigin() helper that normalizes origins before
comparing to window.location.origin. Fixes auto-connect treating
self-referencing replicatedInstances entries as remote connections,
causing duplicate friends/DMs/data. Also hides self-referencing
entries from the Connections panel UI.
populateFromReady() built the federatedId dedup set from ALL existing
DMs, including those belonging to the reconnecting origin. Incoming DMs
then matched their own stale entries and were skipped as "duplicates."
The subsequent origin-removal step deleted the old copies, leaving no
DMs from that origin in state.
Scope the dedup set to DMs from OTHER origins only, so reconnecting
origins replace their DMs cleanly while cross-instance dedup still works.
Three bugs that combined to corrupt DM identities during initial sync:
1. Sync endpoint omitted federatedId for group DMs, causing the receiver
to treat them as 1-on-1 DMs and compute wrong pair hashes — creating
phantom channels that collide with real ones.
2. backfillHomeUserId unconditionally overwrote existing homeUserIds,
so a single wrong match would permanently corrupt a user's identity
and cascade to all subsequent lookups.
3. Migration duplicate-stub Criterion 1 ("shared 1-on-1 DM membership")
incorrectly merged different users from the same domain who were
simply having a conversation, destroying one user's identity.
The relay broadcast loop skipped members whose homeInstance matched the
source instance, assuming they already received the message on their home
server. This broke delivery for federated users (e.g. youruser@nova on
orbit) who are actively connected to the remote instance.
Client-side dedup in addRealtimeMessage already handles double delivery
via sourceMessageId cross-matching, making the server-side skip both
unnecessary and harmful.
Add federation.md section 8b covering dm_close/dm_reopen relay events:
payload, outbound queueing via queueDmCloseRelay, inbound processDmCloseEvent/
processDmReopenEvent handlers (lookup-only identity resolution, silent
no-ops on missing channel/user/membership), and the processCreateEvent
closed-state reopen bug fix.
Update dm-system.md soft-close section with federation behaviour: relay
to peers for both close and reopen, relayed-message reopen in
processCreateEvent, and the federatedId-only guard for legacy DMs.
Jannis Braun