HEALTH_CHECK_INTERVAL_MS was 1 h, but ROTATION_GRACE_PERIOD_MS is 15 min.
Phase skew between two peers' health-check ticks could stretch rotation
finalization desync up to ~1 h, during which signatures from the already-
finalized side verify against the other side's primary-only secret (grace
has expired; verifyPeerSignature stops trying the pending secret). With
AUTH_FAILURE_THRESHOLD = 5 and the existing backoff schedule, this
occasionally tripped legitimate rotations into needs_attention.
Setting the interval to 15 min (= ROTATION_GRACE_PERIOD_MS) guarantees a
finalization tick fires within one grace window on each side, so the
cross-verification window where one peer signs with NEW while the other
still treats NEW as pending cannot outlast the grace period.
Per-tick cost is negligible for the worker's steady state: the only
network fetches are per-active-peer /peer/rotate calls when the 90-day
rotation interval hits (rare) and per-unreachable-peer /instance/info
health pings (bounded by outage count). Going lower than 15 min would
reduce the residual desync but increase tick overhead with diminishing
returns; 15 min is the grace-period-aligned value that the original spec
("runs hourly") deviated from without justification.
Follow-up #20 from S2S DM unification backlog; reduces #19 false-positive
rate (outbox auth-failure transition) on legitimate rotations.
Replace the federation outbox worker's 401/403 wipe-and-rehandshake loop
with bounded retry (AUTH_FAILURE_THRESHOLD=5, ~21.5 min backoff window) and
a new `needs_attention` peer state. Surfaces persistent HMAC desync to
admins via a first-class 'Reset peering' action instead of the prior
silent loop.
Closes backlog item #19. Security invariants verified on live infra
(Pi+VM):
- hmac_secret is NEVER wiped in response to a network-observed 401/403
(Task 5 removes the wipe; Task 7 extends the /peer/accept idempotent-
200-no-update safeguard to cover needs_attention peers).
- Auth failures increment only consecutive_auth_failures, never the
network counter consecutive_failures (Task 5 splits
handleOutboxDeliveryFailure → applyOutboxEntryBackoff).
- Transition occurs at exactly 5 consecutive 401/403 responses; below
threshold, entries get backoff but state is preserved; above, peer
flips to needs_attention, affected users get federation_peer_rejected
WS with 'Federation trust broken — admin must reset peering'.
- /peer/accept safeguard confirmed against attacker curl probe on the
live Pi instance while in needs_attention — forged-secret request
returned 200-no-update, local hmac_secret unchanged.
- Legitimate rotation (Scenario B) does not false-positive: both sides
capture pending secret, auth_failures stays at 0, DM delivers cleanly
during grace period.
Four follow-up backlog items discovered during the work:
#20 health-check cadence tightening (15-min grace vs 1-hour tick)
#21 /peer/initiate 202 handling
#22 consecutive_failures nullability normalization
#23 unify client-side FederationPeer with shared type
Spec: internal notes
Plan: internal notes
Task 11 due-diligence audit for #19. Checked all signed-fetch sites in
packages/server/src for 4xx-branch mutations of federationPeers.hmacSecret
or federationPeers.status:
- sendCallRelay (federationOutbox.ts): on 4xx returns post_failed; on
5xx/network returns peer_transient_failure. No peer-state mutation.
- sendTypingRelay (federationOutbox.ts): delegates to sendCallRelay with
peeringTimeoutMs:0 (fire-and-forget). No peer-state mutation.
- cleanupExpiredApprovalRequests (storageJanitor.ts): sends denial,
only deletes peerApprovalRequests row on success. No federationPeers
mutation.
- DELETE /identity (users.ts): per-origin cleanup; only deletes local
userFederationRegistry on success, never touches federationPeers.
- denyApprovalRequest (federation.ts): requires 2xx from remote before
inserting/updating a rejected peer row. Admin-driven, not wipe.
- POST /peers/:id/rotate (federation.ts): mutates pendingHmacSecret only
on 2xx; returns 502 on 4xx without state change.
- Auto-rotation in federationWorker.ts: same 2xx-gated pattern as manual
rotate.
- Unreachable-recovery health check: only promotes to active on 2xx.
- ensurePeered/performHandshake (federationPeering.ts): on 403 with
PEERING_REQUIRES_APPROVAL sets status='rejected' (explicit, not a
HMAC-mismatch wipe); on other 4xx/5xx only deletes the row if it was
a freshly created placeholder (existingPeerId falsy). Pre-existing
peers are untouched.
Only federationWorker.ts:269 mutates HMAC-related state in response to
401/403, and that path was rewritten in Task 5 to use
evaluateAuthFailure and transition to needs_attention. No additional
handlers require the bounded-retry refactor.
- peerStatusLabel/Color/DotColor gain a 'needs_attention' case (rose).
- StatusFilter row gains 'Needs Attention' toggle.
- PeerRow hides Rotate/Revoke and shows 'Reset Peering' when status is
needs_attention, plus an Auth Failures stat.
- Parent panel routes 'reset' through a ConfirmDialog (danger variant)
that spells out the destructive nature and the out-of-band re-peer step.
- Client FederationPeer interface gains consecutiveAuthFailures (Task 2
extended the shared type but the web client's local mirror was stale).
Codifies the manual 'delete both sides, re-peer' workaround as a
first-class admin action.
Admin-only endpoint for recovering from needs_attention. Deletes the
local peer row; FK cascade removes queued outbox entries. Gated to
peers in needs_attention to prevent accidental resets of healthy
peerings (use /peers/:id for revoke on active peers).
Also extends the Task 8.5 test mock of '../db/index.js' to re-export
`schema`. federation.ts imports `schema` from the re-export alongside
`getDb`; the previous mock only exposed `getDb`, causing the route
handler to blow up with 500s before reaching any assertion. This is
a scaffolding fix — no test assertions were changed.
Four cases from the spec's testing strategy: 404 on missing peer, 400 on
wrong status, 403 for non-admin, and successful delete including FK
cascade of queued outbox entries. Introduces a minimal Fastify-inject
harness for route testing — previously the codebase had only pure-function
unit tests under utils/.
Tests intentionally FAIL at this commit — Task 8 will add the handler and
close the loop.
Unauthenticated /peer/accept must not overwrite hmac_secret for peers in
needs_attention, same as active. needs_attention means 'auth trust broke
and we don't know why' — letting an unauthenticated request flip it back
would reintroduce a path for silent HMAC rotation via the outbox-401 loop
the rest of #19 closes. Legitimate recovery is the admin 'Reset peering'
action (next task).
Pairs with the new 401/403 handler — a 2xx relay confirms HMAC trust is
healthy so the counter should clear. Mirrors the existing
consecutive_failures reset for network-layer health.
Code review of the previous commit found that the backoff branch of the
new 401/403 handler delegated to handleOutboxDeliveryFailure, which
double-dips by also incrementing consecutive_failures (the network-layer
counter that drives the 'unreachable' transition at threshold 10). Per the
design spec §State Machine Changes → Reset logic, auth failures must
increment consecutive_auth_failures ONLY.
Split handleOutboxDeliveryFailure into:
- applyOutboxEntryBackoff: just the per-entry backoff update (safe to call
from the auth-failure path)
- handleOutboxDeliveryFailure: entry backoff + peer's consecutive_failures
bump (network-error path only)
Also adds a console.warn to the backoff branch so operators can diagnose
clock-skew and rotation-grace incidents before the peer hits the terminal
threshold.
Part of backlog #19.
The previous handler (commit ce33ccf + its 403 extension) wiped hmac_secret
and reset peer status to 'pending' on any 401/403 from an active peer. This
collapsed three distinct failure modes — transient clock skew, legitimate
split-brain, active MITM attempt — into "silently establish new trust
immediately." The remote's /peer/accept idempotent-200-no-update safeguard
then prevented the re-handshake from actually working, producing a 1-req/sec
loop observed during backlog #16 verification.
New behavior: increment consecutive_auth_failures, apply backoff to outbox
entries. At AUTH_FAILURE_THRESHOLD (5) transition to needs_attention,
preserve hmac_secret, surface delivery-impossible to affected users,
notify admins. Secret is NEVER wiped in response to a network-observed
401/403.
Part of backlog #19.
Pure function deciding whether the next 401/403 from an active peer
triggers backoff or a transition to needs_attention. Threshold = 5,
corresponding to ~21.5 min of the existing BACKOFF_SCHEDULE_MS.
Adds 'rejected', 'awaiting_approval', and 'needs_attention' to the status
union, plus the consecutiveAuthFailures / autoRotateIntervalDays /
secretRotatedAt / rotationInProgress fields the UI already reads.
Tracks HMAC-failure count separately from consecutive_failures (network
errors). Auth failures and network failures have different resolution
paths; mixing them would let a single successful retry after a network
blip mask real secret desync.
Part of backlog #19 — outbox auth-failure recovery.
Note: drizzle-kit generated the 0003 SQL with an unexpected CREATE TABLE
for peer_approval_requests because the 0002 migration was authored
manually without a corresponding 0002_snapshot.json (see 14041e9). The
generated SQL has been trimmed to the single intended ALTER TABLE. The
regenerated 0003_snapshot.json correctly reflects the full current
schema, so future migrations will diff cleanly.
Closes backlog #16. Implements sendCallRelay/sendTypingRelay auto-peering
and caller-facing dm_call_undeliverable failure surface.
See internal notes
and internal notes
for the full design + implementation plan.
sendFederatedCallStart now collects per-targeted-peer results and emits
a single dm_call_undeliverable event to the caller when any targeted
peer relay fails. Destroys the local ring room when no plausible
recipient remains (no targeted success + no connected local ringee).
LiveKit pre-flight also emits via this path with reason
'livekit_unavailable' instead of a silent console.warn, closing the
60s hang for unconfigured instances.
Guards against phantom toasts when the caller cancels mid-race by
checking getRoom() before emitting.
sendCallRelay now returns CallRelayResult with a typed reason on failure.
When the peer is not already active (or unreachable), runs a racePeering
against CALL_PEERING_TIMEOUT_MS (3s). Background handshake is not aborted
on race loss — next attempt succeeds.
sendTypingRelay passes peeringTimeoutMs:0 so typing never blocks on a
handshake; instead a warm-up ensurePeered runs in the background for any
non-active peer so the NEXT relay (message, call, or typing) benefits.
Addresses code review on b22a7bd: (1) a rejected handshake now returns
{ status: 'failed', error } instead of throwing, keeping the structured
contract; (2) the "background handshake" warn only fires when the
timeout arm wins — not when the handshake is itself the race winner by
rejection. Timing tests migrated to vi.useFakeTimers for determinism.
Regression test added for the handshake-wins-by-rejection case.
Exports `racePeering(origin, timeoutMs, ensurePeeredFn?)` that races
`ensurePeered` against a deadline. On timeout, the background handshake
continues (warming the peer for the next attempt) and a warn-logged
.catch() prevents unhandledRejection. Injectable `ensurePeeredFn` param
enables full DI in tests without mocking module internals.
processMemberAddEvent, processMemberRemoveEvent, and processOwnershipTransferEvent inserted system messages unconditionally. Outbox retries and initial-sync replays (triggered whenever an admin re-approves a peering request, which recreates the peer row with lastSyncedAt=0) duplicated the system message on every delivery. Each new snowflake ID exceeded the user's last_read_message_id, flipping the channel back to unread after every deploy.
Processors now short-circuit on a matching (source_instance, source_message_id) row, and persist those fields when inserting. processMemberAddEvent emits the tagged system message in both bootstrap and incremental paths so bootstrap replays don't fall through and insert a second one; the bootstrap's dm_channel_created broadcast carries that message as lastMessage so sidebar previews and unread anchors agree across instances.
When two instances each have a native user with the same username, the
Add Friend search card for the federated one sent its request to the
local namesake instead of the intended remote user.
Root cause: `isNative = !homeUserId` in socialStore's searchUsers and
loadFriends dedup. The server backfills native users' homeUserId to
their own id so federation tier-1 lookups succeed, so `homeUserId` is
set on natives too. Only `homeInstance` distinguishes native (null)
from replicated stubs. With the wrong check, no entry was ever "native"
and the home-origin stub of the remote user was kept over the true
native record — leaving `_instanceOrigin=''`, which caused the Send
button handler to drop the domain suffix and POST to the home API,
where "nova" resolved to a completely different local user.
Also fixes loadRequests dedup to prefer the target-native record so the
search card correctly flips to "Request Pending" after sending.
Added federation_peers_changed (no-payload signal) broadcast from every
peer state mutation, and federation_approval_request_received when a new
approval request is queued. Client subscribes via onFederationPeersChanged
callback registry. FederationPanel and PendingApprovals debounce-refetch
on any event. sendToAdmins helper broadcasts only to admin users.
autoAcceptPeering means 'don't accept peering initiated by others', not
'don't initiate peering ourselves'. Two checks were incorrectly blocking
outgoing peering when auto-accept was off:
1. ensurePeered() refused to auto-initiate — reverted. When a local user
sends a DM, the server should initiate peering. The remote's
peer/accept decides whether to accept or queue.
2. queueOutboxEvent() refused to create placeholders — reverted. The
outbox needs placeholders to queue entries. Without them, DM relay
silently fails.
When both instances have autoAcceptPeering off, the approval flow
ping-ponged indefinitely. Admin A approves → handshakes to B → B
queues (202) → A's peer becomes awaiting_approval. Admin B approves →
handshakes to A → but A's gate only matched 'pending', not
'awaiting_approval', so it re-queued instead of accepting.
Now the gate matches both 'pending' and 'awaiting_approval'. When the
second admin approves and handshakes back, the first instance recognizes
its admin already approved and accepts — completing the peering.
1. queueOutboxEvent no longer creates pending peer placeholders when
autoAcceptPeering is disabled — prevents bypassing the admin's
peering control
2. Approval endpoint checks for 202 before response.ok — when the
remote also has autoAcceptPeering off, sets peer to awaiting_approval
instead of incorrectly activating it
3. awaiting_approval status added to Federation panel UI — status label,
colors, filter options so these peers are visible and manageable
When all peers are pending (no active peers with outbox entries),
processOutboxTick returned early at line 141 before reaching
resolvePendingPeers at line 302. Pending peers were never resolved
because the only code path to resolvePendingPeers was after the
active-peer delivery loop — which never ran.
getGroupDmTargetOrigins() returned undefined for 1-on-1 DMs, which
queueOutboxEvent() treated as 'broadcast to all existing peers'. When
no peers existed, nothing was queued and no handshake was ever triggered.
Now always computes target origins from DM participants so the pending
placeholder creation path runs, enabling ensurePeered() → peer/accept
→ approval queue flow.
The client's direct WS connection to remote instances (via Connections)
delivered DM events independently of S2S peering. Added activePeerOrigins
allowlist to ready payload — all DM event handlers now silently drop
events from non-home origins without an active peer. This prevents
notifications, sounds, previews, typing indicators, calls, and channel
updates from instances where peering was revoked or never established.
ensurePeered() now checks the local autoAcceptPeering setting before
initiating new peering. When disabled, only admin-explicit peer/initiate
and approval-request approve bypass this check. Closes the bypass where
client peer/ensure or outbox worker could auto-initiate outward peering
even when the admin intended to control all peering.
Jannis Braun