The outbox worker rebuilds FederationRelayEvent objects from stored JSON via
an allowlist of known fields. presenceUpdate was missed when presence_update
events shipped, so peers received events with eventType='presence_update' but
no payload — rejected with missing_presence_update_payload on every tick.
hydrateReplicatedUserProfile now calls downloadProfileAsset and stores bare local filenames, falling back to absolute URLs only on download failure. It also fills empty fields only — no longer clobbering local files written by processProfileUpdateEvent. Adds an idempotent startup backfill that converts existing http-prefixed avatar/banner rows on replicated users into local files, so federated profile pictures keep rendering when the home instance is offline.
Two correctness/defense fixes plus regression tests in the existing
in-memory drizzle test file.
1. Reverse-direction idempotency. The sender-side path in social.ts
checks BOTH directions of friend_requests and returns 409
incoming_request_exists when an opposite-direction row exists. The
receiver only matched from->to, so cross-fire (alice@A and bob@B both
click "add friend" near-simultaneously) produced two opposite
pending rows on each instance. The receiver now silent-accepts when
either direction matches a pending row, mirroring the sender's
both-direction check.
2. Self-target guard (defense-in-depth). Reject events whose
from-identity equals to-identity (after normalizeOriginForCompare)
with a new receiver-acknowledged 4xx code self_target_invalid.
Sender's local cannot_friend_self should catch this, but the
receiver does not trust upstream validation. Added to
TERMINAL_REJECTION_REASONS so the standard rollback fires
(mapped client-side to peer_rejected). Logged at console.warn.
Spec updates: social.md inbound contract now documents both-direction
idempotency and the self-target guard; federation.md and the
s2s-friend-add design spec list the new terminal rejection reason.
- social.ts friend-add: user_action, with 409 peer_pending_local_admin
when gate fires
- /peer/ensure: user_action, surfaces peeringStatus: 'admin_required'
- sendCallRelay (typing warm-up + call relay): system intent
- federationWorker resolvePendingPeers: system intent (defensive — gate
is unreachable from here since pending rows already exist)
- CallRelayFailureReason: peer_admin_required added (mapped to
peer_transient_failure on the user-facing event surface, since system
intent should never legitimately surface admin_required)
- Test files: thread intent arg through racePeering and ensurePeered
calls (positional shift from racePeering signature change)
- outboundGate.test.ts: tighten noUncheckedIndexedAccess access via
non-null assertions after toHaveLength()
- docs/systems/social.md: peer_pending_local_admin error code documented
Duplicate rejection means the peer already has the message (e.g.,
delivered earlier via outbox AND pulled via sync in the same
window). Retrying will fail identically forever until TTL expires.
Before this patch: duplicate-rejected outbox entries were retained
with attempts++ and exponential backoff, creating log noise and
outbox bloat for up to 30 days.
After: duplicate-rejected entityIds join the terminal set alongside
accepted ones and are deleted from the outbox. Logged at info level
('outbox entry removed (terminal)') to distinguish from warn-level
transient-rejection retries.
Other rejection reasons (attribution_mismatch, processing_error,
etc.) stay on the retry path; some may also be terminal but are
deferred until observed accumulating.
Four event types previously bypassed appendMutationLog, making
them unrecoverable via /api/federation/sync after peer inactivity:
- queueDmCloseRelay (dm_close, dm_reopen)
- queueReadStateRelay (read_state_update)
- handleSizeRejection in federationWorker (file_rejected)
- profile PATCH route (profile_update) — two call sites,
one appendMutationLog per profile change (not per target origin)
The /api/federation/sync response builder is extended to
serialize these event types in the next task.
Every code location that sets federation_peers.status='active'
now invokes onPeerActivated(peerId, reason). HTTP handler sites
use fire-and-forget (.catch(log)) so the response isn't blocked
by sync-pull pagination. The worker-internal health-check site
awaits the handler since the tick is already async.
Sites: /peer/initiate, /peer/accept (4 branches), /approval-
requests/:id/approve, health check recovery, ensurePeered/
performHandshake.
Missed in 02a1ed7. The new sync-pull path in federationPeerActivation.ts
uses a dynamic import of processRelayEvents from routes/federation.js;
the static import in federationWorker.ts is no longer used after
runInitialSyncForNewPeers deletion.
The per-peer sync body is now syncPeerMutationLog (in the new
peer-activation module), invoked via onPeerActivated. The startup
path scans for status='active' AND lastSyncedAt=0 and calls the
unified handler for each — same trigger condition as before, unified
code path with runtime transitions.
HEALTH_CHECK_INTERVAL_MS was 1 h, but ROTATION_GRACE_PERIOD_MS is 15 min.
Phase skew between two peers' health-check ticks could stretch rotation
finalization desync up to ~1 h, during which signatures from the already-
finalized side verify against the other side's primary-only secret (grace
has expired; verifyPeerSignature stops trying the pending secret). With
AUTH_FAILURE_THRESHOLD = 5 and the existing backoff schedule, this
occasionally tripped legitimate rotations into needs_attention.
Setting the interval to 15 min (= ROTATION_GRACE_PERIOD_MS) guarantees a
finalization tick fires within one grace window on each side, so the
cross-verification window where one peer signs with NEW while the other
still treats NEW as pending cannot outlast the grace period.
Per-tick cost is negligible for the worker's steady state: the only
network fetches are per-active-peer /peer/rotate calls when the 90-day
rotation interval hits (rare) and per-unreachable-peer /instance/info
health pings (bounded by outage count). Going lower than 15 min would
reduce the residual desync but increase tick overhead with diminishing
returns; 15 min is the grace-period-aligned value that the original spec
("runs hourly") deviated from without justification.
Follow-up #20 from S2S DM unification backlog; reduces #19 false-positive
rate (outbox auth-failure transition) on legitimate rotations.
Pairs with the new 401/403 handler — a 2xx relay confirms HMAC trust is
healthy so the counter should clear. Mirrors the existing
consecutive_failures reset for network-layer health.
Code review of the previous commit found that the backoff branch of the
new 401/403 handler delegated to handleOutboxDeliveryFailure, which
double-dips by also incrementing consecutive_failures (the network-layer
counter that drives the 'unreachable' transition at threshold 10). Per the
design spec §State Machine Changes → Reset logic, auth failures must
increment consecutive_auth_failures ONLY.
Split handleOutboxDeliveryFailure into:
- applyOutboxEntryBackoff: just the per-entry backoff update (safe to call
from the auth-failure path)
- handleOutboxDeliveryFailure: entry backoff + peer's consecutive_failures
bump (network-error path only)
Also adds a console.warn to the backoff branch so operators can diagnose
clock-skew and rotation-grace incidents before the peer hits the terminal
threshold.
Part of backlog #19.
The previous handler (commit ce33ccf + its 403 extension) wiped hmac_secret
and reset peer status to 'pending' on any 401/403 from an active peer. This
collapsed three distinct failure modes — transient clock skew, legitimate
split-brain, active MITM attempt — into "silently establish new trust
immediately." The remote's /peer/accept idempotent-200-no-update safeguard
then prevented the re-handshake from actually working, producing a 1-req/sec
loop observed during backlog #16 verification.
New behavior: increment consecutive_auth_failures, apply backoff to outbox
entries. At AUTH_FAILURE_THRESHOLD (5) transition to needs_attention,
preserve hmac_secret, surface delivery-impossible to affected users,
notify admins. Secret is NEVER wiped in response to a network-observed
401/403.
Part of backlog #19.
Added federation_peers_changed (no-payload signal) broadcast from every
peer state mutation, and federation_approval_request_received when a new
approval request is queued. Client subscribes via onFederationPeersChanged
callback registry. FederationPanel and PendingApprovals debounce-refetch
on any event. sendToAdmins helper broadcasts only to admin users.
When all peers are pending (no active peers with outbox entries),
processOutboxTick returned early at line 141 before reaching
resolvePendingPeers at line 302. Pending peers were never resolved
because the only code path to resolvePendingPeers was after the
active-peer delivery loop — which never ran.
When a profile_update relay arrives with avatar/banner URLs, download
the files to local storage instead of storing remote absolute URLs.
Falls back to absolute URL on any download failure. Cleans up old
local files when replaced.
The outbox worker's event reconstruction whitelist was missing these two
fields, causing read_state_update and dm_close/dm_reopen relay events to
arrive at remote instances with empty payloads and get rejected.
- Fix race window: store pendingHmacSecret AFTER remote peer confirms,
not before (admin endpoint + auto-rotation worker)
- Add hex validation on newSecret at /peer/rotate endpoint
- Use pending-secret-aware signing in initial sync worker
- Add test for corrupt state (pendingHmacSecret set, secretRotationAt null)
Extract processRelayEvents() from the relay HTTP handler and call it
directly in runInitialSyncForNewPeers(), eliminating the HTTP round-trip
through public DNS that failed on networks without hairpin NAT.
Three stacked bugs prevented federation relay from working for group DMs:
1. Origin format mismatch: users.home_instance stores bare domains
("nova.ddns.net") but federation_peers.origin stores full URLs
("https://nova.ddns.net"). getGroupDmTargetOrigins() built target
lists from bare domains, so queueOutboxEvent() never matched any
peers — events were never queued into the outbox.
2. Missing federatedId in outbox reconstruction: the outbox worker
rebuilt relay events from stored payloads but never copied the
federatedId field. Receiving instances check this field and rejected
all member_add/remove/ownership_transfer events with
"missing_membership_payload".
3. Duplicate channels from cross-instance broadcasts: dm_channel_created
was sent to ALL members including remote replicas. Users connected to
multiple instances received the event twice (once per instance),
creating duplicate group DMs in their sidebar. Fixed by only
broadcasting to members whose home instance matches the current
server — remote members receive the channel via federation bootstrap
on their home instance.
- Fix critical: outbox worker now copies file_rejected payload fields
(attachmentId, sourceFilename, rejectionReason, rejectionLimit,
affectedUserIds) so the reverse relay actually delivers them
- Fix: add sourceFilename to file_rejected event for reliable
multi-attachment matching on the sender side
- Fix: change text-accent-warning to text-accent-amber (valid class)
- Add Array.isArray guard on federationMeta parse
Replace the hard-delete in the leave handler with a soft-delete (sets
deleted_at timestamp) when the last member leaves a group DM. A new
janitor sweep in the federation worker runs hourly and purges channels
whose grace period has expired, cascading through reactions, embeds,
attachments, messages, members, outbox/mutation-log/file-queue entries,
and finally the channel itself.
All client-facing dm_channels queries now filter on deleted_at IS NULL
to hide soft-deleted channels from the REST API and WebSocket ready
payload.
Consolidates 4 inline constructions of the instance origin URL into a
single shared function. Removes the private copy in federationWorker
and two ad-hoc domainOrigin variables in federationOutbox.
After the file worker downloads a federated attachment, generate a
thumbnail using the same sharp pipeline as local uploads. This ensures
federated images use bandwidth-efficient thumbnails in chat view
instead of serving the full-size image.
Also fixes processCreateEvent to not copy the source instance's
thumbnailFilename (which doesn't exist locally).
The file worker now UPDATEs the attachment row created by processCreateEvent
(swapping sourceUrl interim filename to local path) instead of INSERTing a
duplicate. Falls back to INSERT for legacy queue entries.
After download, broadcasts dm_message_updated so clients see the attachment
transition from remote hotlink to local file in real-time.
The relay was failing because processCreateEvent relied on the friends
table to discover the DM recipient, but friendships aren't federated
across instances. Also, resolveLocalUser matched deleted replicated
users before active ones.
- Add participants[] to FederationRelayEvent with homeUserId/homeInstance
for all DM channel members
- Add getDmParticipants() helper to look up member identities
- Include participants in outbox payloads (create/update) and sync events
- Rewrite processCreateEvent to resolve participants directly, compute
canonicalDmPairId, and findOrCreateDmChannel — removing the entire
friends-list fallback (60+ lines)
- Fix resolveLocalUser to filter out deleted users (is_deleted = 0)
and prefer the replicated user match when multiple candidates exist
Jannis Braun