Four event types previously bypassed appendMutationLog, making
them unrecoverable via /api/federation/sync after peer inactivity:
- queueDmCloseRelay (dm_close, dm_reopen)
- queueReadStateRelay (read_state_update)
- handleSizeRejection in federationWorker (file_rejected)
- profile PATCH route (profile_update) — two call sites,
one appendMutationLog per profile change (not per target origin)
The /api/federation/sync response builder is extended to
serialize these event types in the next task.
Every code location that sets federation_peers.status='active'
now invokes onPeerActivated(peerId, reason). HTTP handler sites
use fire-and-forget (.catch(log)) so the response isn't blocked
by sync-pull pagination. The worker-internal health-check site
awaits the handler since the tick is already async.
Sites: /peer/initiate, /peer/accept (4 branches), /approval-
requests/:id/approve, health check recovery, ensurePeered/
performHandshake.
/peer/initiate checked `response.ok` to decide whether to activate the
local peer. `response.ok` is true for the full 2xx range, so a remote
that returned 202 (queued for admin approval — autoAcceptPeering off
on their side) caused the local peer to flip to `active` while the
remote had us `awaiting_approval`. The split only self-healed when
the remote admin approved and pushed us an `awaiting_approval → active`
override via the peer_approval_requests inbound path.
The auto-peer flow in federationPeering.ts:performHandshake already
had the correct 202 branch: set local status to awaiting_approval,
broadcast federation_peers_changed, surface a pending outcome. Mirror
it here:
- Check response.status === 202 BEFORE the !response.ok branch so the
fall-through can't reach the activation code.
- Transition local peer to awaiting_approval (not active).
- Broadcast federation_peers_changed so other admin tabs refresh.
- Return 202 with the sanitized peer so the client observes the
queued state distinctly from both success and failure.
Also added the missing federation_peers_changed broadcast on the
activation (200) path for parity with every other peer-state-change
site in the codebase — it was a pre-existing drift that would leave
sibling admin tabs stale after an initiate. Pattern-aligned with
federationPeering.ts:160 and the rest of routes/federation.ts.
Docs: expanded Phase 1 bullets in docs/systems/federation.md to cover
the 200 / 202 / other non-2xx / network-error branches explicitly and
reference the mirrored auto-peer branch.
Verified: pnpm -r typecheck clean (shared + server), vitest 70/70
pass.
Closes#21 from S2S DM unification backlog.
The column was `integer DEFAULT 0` (nullable) since the initial schema.
Counters should not be nullable — the semantics are a count, not an
optional measurement. `consecutive_auth_failures` (added later) was
correctly declared NOT NULL; tightening `consecutive_failures` to match
removes the drift and eliminates the "|null" burden everywhere the value
is read.
SQLite does not support in-place ALTER … SET NOT NULL, so drizzle-kit
cannot auto-generate this. The manual migration uses the standard
SQLite recreate pattern (new table + INSERT SELECT + DROP + RENAME +
recreate index) under `PRAGMA defer_foreign_keys = ON` so the existing
federation_outbox → federation_peers FK survives the swap. The COPY
step coalesces any hypothetical NULL to 0 defensively; live probes on
both test instances (nova, orbit) showed zero NULL rows so no
actual backfill is required.
Verified by applying the full migration chain against a copy of the VM's
live DB: column ends as `notnull=1 dflt=0`, the peer row is preserved,
the unique index on origin is recreated, NULL inserts are rejected, and
`PRAGMA foreign_key_check` reports no violations.
Server `SanitizedPeer.consecutiveFailures` tightened to `number` to
match the new drizzle inference and the shared `FederationPeer` shape.
Follow-up #22 from S2S DM unification backlog.
Admin-only endpoint for recovering from needs_attention. Deletes the
local peer row; FK cascade removes queued outbox entries. Gated to
peers in needs_attention to prevent accidental resets of healthy
peerings (use /peers/:id for revoke on active peers).
Also extends the Task 8.5 test mock of '../db/index.js' to re-export
`schema`. federation.ts imports `schema` from the re-export alongside
`getDb`; the previous mock only exposed `getDb`, causing the route
handler to blow up with 500s before reaching any assertion. This is
a scaffolding fix — no test assertions were changed.
Four cases from the spec's testing strategy: 404 on missing peer, 400 on
wrong status, 403 for non-admin, and successful delete including FK
cascade of queued outbox entries. Introduces a minimal Fastify-inject
harness for route testing — previously the codebase had only pure-function
unit tests under utils/.
Tests intentionally FAIL at this commit — Task 8 will add the handler and
close the loop.
Unauthenticated /peer/accept must not overwrite hmac_secret for peers in
needs_attention, same as active. needs_attention means 'auth trust broke
and we don't know why' — letting an unauthenticated request flip it back
would reintroduce a path for silent HMAC rotation via the outbox-401 loop
the rest of #19 closes. Legitimate recovery is the admin 'Reset peering'
action (next task).
processMemberAddEvent, processMemberRemoveEvent, and processOwnershipTransferEvent inserted system messages unconditionally. Outbox retries and initial-sync replays (triggered whenever an admin re-approves a peering request, which recreates the peer row with lastSyncedAt=0) duplicated the system message on every delivery. Each new snowflake ID exceeded the user's last_read_message_id, flipping the channel back to unread after every deploy.
Processors now short-circuit on a matching (source_instance, source_message_id) row, and persist those fields when inserting. processMemberAddEvent emits the tagged system message in both bootstrap and incremental paths so bootstrap replays don't fall through and insert a second one; the bootstrap's dm_channel_created broadcast carries that message as lastMessage so sidebar previews and unread anchors agree across instances.
Added federation_peers_changed (no-payload signal) broadcast from every
peer state mutation, and federation_approval_request_received when a new
approval request is queued. Client subscribes via onFederationPeersChanged
callback registry. FederationPanel and PendingApprovals debounce-refetch
on any event. sendToAdmins helper broadcasts only to admin users.
When both instances have autoAcceptPeering off, the approval flow
ping-ponged indefinitely. Admin A approves → handshakes to B → B
queues (202) → A's peer becomes awaiting_approval. Admin B approves →
handshakes to A → but A's gate only matched 'pending', not
'awaiting_approval', so it re-queued instead of accepting.
Now the gate matches both 'pending' and 'awaiting_approval'. When the
second admin approves and handshakes back, the first instance recognizes
its admin already approved and accepts — completing the peering.
1. queueOutboxEvent no longer creates pending peer placeholders when
autoAcceptPeering is disabled — prevents bypassing the admin's
peering control
2. Approval endpoint checks for 202 before response.ok — when the
remote also has autoAcceptPeering off, sets peer to awaiting_approval
instead of incorrectly activating it
3. awaiting_approval status added to Federation panel UI — status label,
colors, filter options so these peers are visible and manageable
- C1: Include 'unreachable' peers in queueOutboxEvent query to prevent
UNIQUE constraint violation when creating placeholders
- I1: Add 'rejected' to StatusFilter in FederationPanel so admins can
see and manage rejected peers with delete/re-initiate actions
- I2: Map ensurePeered 'failed' to 'pending' in peer/ensure response
to match spec and client expectations
Remove the owner-only gate on POST /api/dm/:id/members. The S2S relay
already accepts member_add from any HMAC-verified peer, and the UI
already shows the add button to all group DM members. Only the
server-side check was blocking non-owners.
When a profile_update relay arrives with avatar/banner URLs, download
the files to local storage instead of storing remote absolute URLs.
Falls back to absolute URL on any download failure. Cleans up old
local files when replaced.
Four fixes addressing the full state management problem:
1. Passive ready handler: no longer auto-connects to LiveKit on
page refresh. Prevents identity conflicts when the same user
has multiple sessions fighting for one LiveKit identity slot.
The user must re-accept to join; state is shown but not acted on.
2. SoundController sync guard: incomingCallLoading/outgoingCallLoading
refs prevent multiple playSound calls during async audio load.
If call is cancelled while sound loads, stops it immediately on
completion. Eliminates the "5 ringtones at once" bug.
3. Host dm_call_accepted broadcasts now include federatedCallId so
all clients (including remote instances) can match the event.
4. Removed all diagnostic console.log statements.
Three fixes for multi-instance call state consistency:
1. Client dm_call_accepted handler only auto-connects to LiveKit if
the user was the caller (outgoingCall was set). Other instances of
the same user just clear ringing state without connecting.
2. Server processDmCallAcceptEvent remote path skips duplicate
broadcast when FederatedCallEntry is already active (prevents
state conflicts from host fan-out arriving after local accept).
3. Ready payload handler clears stuck incomingCall when restoring
an already-active call after page refresh.
When findOrCreateDmChannel creates or finds a local channel for a
federatedId that has an active FederatedCallEntry with null dmChannelId,
update the entry. Prevents stale null references for Path B calls.
Three bugs that combined to corrupt DM identities during initial sync:
1. Sync endpoint omitted federatedId for group DMs, causing the receiver
to treat them as 1-on-1 DMs and compute wrong pair hashes — creating
phantom channels that collide with real ones.
2. backfillHomeUserId unconditionally overwrote existing homeUserIds,
so a single wrong match would permanently corrupt a user's identity
and cascade to all subsequent lookups.
3. Migration duplicate-stub Criterion 1 ("shared 1-on-1 DM membership")
incorrectly merged different users from the same domain who were
simply having a conversation, destroying one user's identity.
The relay broadcast loop skipped members whose homeInstance matched the
source instance, assuming they already received the message on their home
server. This broke delivery for federated users (e.g. youruser@nova on
orbit) who are actively connected to the remote instance.
Client-side dedup in addRealtimeMessage already handles double delivery
via sourceMessageId cross-matching, making the server-side skip both
unnecessary and harmful.
Queue dm_close relay in DELETE /api/dm/:id after broadcasting locally,
and queue dm_reopen relay inside the myDm.closed === 1 branch of POST
/api/dm so peers are notified only on actual reopen, not new DM creation.
Receive read_state_update events from peers, translate message
coordinates to local IDs via sourceInstance/sourceMessageId
mapping, update read_states with timestamp-only LWW.
profileUpdatedAt ?? createdAt treated freshly registered users as having
"newer" profiles than users with real edit history, because createdAt is
always NOW at registration time. This broke federation profile sync:
the client correctly pushed home → remote, but the remote server's LWW
guard rejected the write (stored createdAt > incoming profileUpdatedAt).
A null profileUpdatedAt means "never edited" — that's timestamp 0, not
the user's creation time.
After deleting a federated identity, the server-side user_federation_registry
and users.replicated_instances were not cleaned up, causing "already connected"
errors when trying to re-federate. The deletion endpoint now authoritatively
removes both the registry row and the replicatedInstances entry, and bumps the
LWW timestamp to prevent stale client syncs from re-inserting them.
Also extends the endpoint to accept mode 'leave' (skip S2S, just clean up),
and enables the "Select instances..." scope option in DeleteIdentityDialog.
Switch from manual space-ID collection to collectDeletionBroadcastTargets and
add user_updated broadcast so clients patch their caches when a federated user
is deleted via S2S. Force-disconnect moved after broadcasts so other tabs
receive events before the connection is torn down.
After a prior deletion + re-federation, multiple user records share
the same homeUserId (one deleted, one live). The unfiltered .get()
returned the older deleted record, causing the idempotency check to
short-circuit and miss the live record entirely.
Jannis Braun