HEALTH_CHECK_INTERVAL_MS was 1 h, but ROTATION_GRACE_PERIOD_MS is 15 min.
Phase skew between two peers' health-check ticks could stretch rotation
finalization desync up to ~1 h, during which signatures from the already-
finalized side verify against the other side's primary-only secret (grace
has expired; verifyPeerSignature stops trying the pending secret). With
AUTH_FAILURE_THRESHOLD = 5 and the existing backoff schedule, this
occasionally tripped legitimate rotations into needs_attention.
Setting the interval to 15 min (= ROTATION_GRACE_PERIOD_MS) guarantees a
finalization tick fires within one grace window on each side, so the
cross-verification window where one peer signs with NEW while the other
still treats NEW as pending cannot outlast the grace period.
Per-tick cost is negligible for the worker's steady state: the only
network fetches are per-active-peer /peer/rotate calls when the 90-day
rotation interval hits (rare) and per-unreachable-peer /instance/info
health pings (bounded by outage count). Going lower than 15 min would
reduce the residual desync but increase tick overhead with diminishing
returns; 15 min is the grace-period-aligned value that the original spec
("runs hourly") deviated from without justification.
Follow-up #20 from S2S DM unification backlog; reduces #19 false-positive
rate (outbox auth-failure transition) on legitimate rotations.
Pairs with the new 401/403 handler — a 2xx relay confirms HMAC trust is
healthy so the counter should clear. Mirrors the existing
consecutive_failures reset for network-layer health.
Code review of the previous commit found that the backoff branch of the
new 401/403 handler delegated to handleOutboxDeliveryFailure, which
double-dips by also incrementing consecutive_failures (the network-layer
counter that drives the 'unreachable' transition at threshold 10). Per the
design spec §State Machine Changes → Reset logic, auth failures must
increment consecutive_auth_failures ONLY.
Split handleOutboxDeliveryFailure into:
- applyOutboxEntryBackoff: just the per-entry backoff update (safe to call
from the auth-failure path)
- handleOutboxDeliveryFailure: entry backoff + peer's consecutive_failures
bump (network-error path only)
Also adds a console.warn to the backoff branch so operators can diagnose
clock-skew and rotation-grace incidents before the peer hits the terminal
threshold.
Part of backlog #19.
The previous handler (commit ce33ccf + its 403 extension) wiped hmac_secret
and reset peer status to 'pending' on any 401/403 from an active peer. This
collapsed three distinct failure modes — transient clock skew, legitimate
split-brain, active MITM attempt — into "silently establish new trust
immediately." The remote's /peer/accept idempotent-200-no-update safeguard
then prevented the re-handshake from actually working, producing a 1-req/sec
loop observed during backlog #16 verification.
New behavior: increment consecutive_auth_failures, apply backoff to outbox
entries. At AUTH_FAILURE_THRESHOLD (5) transition to needs_attention,
preserve hmac_secret, surface delivery-impossible to affected users,
notify admins. Secret is NEVER wiped in response to a network-observed
401/403.
Part of backlog #19.
Added federation_peers_changed (no-payload signal) broadcast from every
peer state mutation, and federation_approval_request_received when a new
approval request is queued. Client subscribes via onFederationPeersChanged
callback registry. FederationPanel and PendingApprovals debounce-refetch
on any event. sendToAdmins helper broadcasts only to admin users.
When all peers are pending (no active peers with outbox entries),
processOutboxTick returned early at line 141 before reaching
resolvePendingPeers at line 302. Pending peers were never resolved
because the only code path to resolvePendingPeers was after the
active-peer delivery loop — which never ran.
When a profile_update relay arrives with avatar/banner URLs, download
the files to local storage instead of storing remote absolute URLs.
Falls back to absolute URL on any download failure. Cleans up old
local files when replaced.
The outbox worker's event reconstruction whitelist was missing these two
fields, causing read_state_update and dm_close/dm_reopen relay events to
arrive at remote instances with empty payloads and get rejected.
- Fix race window: store pendingHmacSecret AFTER remote peer confirms,
not before (admin endpoint + auto-rotation worker)
- Add hex validation on newSecret at /peer/rotate endpoint
- Use pending-secret-aware signing in initial sync worker
- Add test for corrupt state (pendingHmacSecret set, secretRotationAt null)
Extract processRelayEvents() from the relay HTTP handler and call it
directly in runInitialSyncForNewPeers(), eliminating the HTTP round-trip
through public DNS that failed on networks without hairpin NAT.
Three stacked bugs prevented federation relay from working for group DMs:
1. Origin format mismatch: users.home_instance stores bare domains
("nova.ddns.net") but federation_peers.origin stores full URLs
("https://nova.ddns.net"). getGroupDmTargetOrigins() built target
lists from bare domains, so queueOutboxEvent() never matched any
peers — events were never queued into the outbox.
2. Missing federatedId in outbox reconstruction: the outbox worker
rebuilt relay events from stored payloads but never copied the
federatedId field. Receiving instances check this field and rejected
all member_add/remove/ownership_transfer events with
"missing_membership_payload".
3. Duplicate channels from cross-instance broadcasts: dm_channel_created
was sent to ALL members including remote replicas. Users connected to
multiple instances received the event twice (once per instance),
creating duplicate group DMs in their sidebar. Fixed by only
broadcasting to members whose home instance matches the current
server — remote members receive the channel via federation bootstrap
on their home instance.
- Fix critical: outbox worker now copies file_rejected payload fields
(attachmentId, sourceFilename, rejectionReason, rejectionLimit,
affectedUserIds) so the reverse relay actually delivers them
- Fix: add sourceFilename to file_rejected event for reliable
multi-attachment matching on the sender side
- Fix: change text-accent-warning to text-accent-amber (valid class)
- Add Array.isArray guard on federationMeta parse
Replace the hard-delete in the leave handler with a soft-delete (sets
deleted_at timestamp) when the last member leaves a group DM. A new
janitor sweep in the federation worker runs hourly and purges channels
whose grace period has expired, cascading through reactions, embeds,
attachments, messages, members, outbox/mutation-log/file-queue entries,
and finally the channel itself.
All client-facing dm_channels queries now filter on deleted_at IS NULL
to hide soft-deleted channels from the REST API and WebSocket ready
payload.
Consolidates 4 inline constructions of the instance origin URL into a
single shared function. Removes the private copy in federationWorker
and two ad-hoc domainOrigin variables in federationOutbox.
After the file worker downloads a federated attachment, generate a
thumbnail using the same sharp pipeline as local uploads. This ensures
federated images use bandwidth-efficient thumbnails in chat view
instead of serving the full-size image.
Also fixes processCreateEvent to not copy the source instance's
thumbnailFilename (which doesn't exist locally).
The file worker now UPDATEs the attachment row created by processCreateEvent
(swapping sourceUrl interim filename to local path) instead of INSERTing a
duplicate. Falls back to INSERT for legacy queue entries.
After download, broadcasts dm_message_updated so clients see the attachment
transition from remote hotlink to local file in real-time.
The relay was failing because processCreateEvent relied on the friends
table to discover the DM recipient, but friendships aren't federated
across instances. Also, resolveLocalUser matched deleted replicated
users before active ones.
- Add participants[] to FederationRelayEvent with homeUserId/homeInstance
for all DM channel members
- Add getDmParticipants() helper to look up member identities
- Include participants in outbox payloads (create/update) and sync events
- Rewrite processCreateEvent to resolve participants directly, compute
canonicalDmPairId, and findOrCreateDmChannel — removing the entire
friends-list fallback (60+ lines)
- Fix resolveLocalUser to filter out deleted users (is_deleted = 0)
and prefer the replicated user match when multiple candidates exist
Jannis Braun