Initiator side of the bidirectional handshake exchange. /peer/accept now returns instanceName in the response body (prior commit); performHandshake parses it and persists alongside status='active'. Tolerates missing field (older peers) and non-JSON bodies.
Follow-up from Task 2 code review. Keeps the test mock aligned with
the widened return type even though vi.mock doesn't structurally
typecheck the factory.
sendFederatedCallStart now treats a 200-with-undeliverable-messageId as
a peer-failure instead of unconditional success. Feeds the existing
failures[] array and terminal-determination machinery from #16.
New sendFederatedCallStartForTest export mirrors the existing
handleDm*ForTest pattern. TDD — three tests cover single-peer terminal
no_recipient, group-DM mixed delivered+undeliverable non-terminal, and
the happy-path (empty undeliverable → no event).
Also hardens sendCallRelay's response parse: validates undeliverable
is an Array and entries are well-shaped, logs protocol drift at warn/debug
rather than silently falling back to old-peer semantics.
CallRelayResult success arm gains undeliverable: string[]. sendCallRelay
parses FederationRelayResponse.undeliverable (when present) and returns
the messageIds so sendFederatedCallStart can reclassify per-peer results.
Old peers that omit the field → empty array → today's behavior.
TDD — three tests cover old-peer, new-peer-with-undeliverable, and 5xx paths.
The admin reset endpoint (DELETE-pattern gated on peer.status !== 'needs_attention')
doesn't transition status — it deletes the row of an already-deactivated peer.
onPeerDeactivated already fired at the earlier needs_attention transition, so
the reset site correctly has no hook. The enum value was defensive-unused; per
project principles (no backwards-compat shims, no placeholders) drop it.
Duplicate rejection means the peer already has the message (e.g.,
delivered earlier via outbox AND pulled via sync in the same
window). Retrying will fail identically forever until TTL expires.
Before this patch: duplicate-rejected outbox entries were retained
with attempts++ and exponential backoff, creating log noise and
outbox bloat for up to 30 days.
After: duplicate-rejected entityIds join the terminal set alongside
accepted ones and are deleted from the outbox. Logged at info level
('outbox entry removed (terminal)') to distinguish from warn-level
transient-rejection retries.
Other rejection reasons (attribution_mismatch, processing_error,
etc.) stay on the retry path; some may also be terminal but are
deferred until observed accumulating.
Replace the batch-level processRelayEvents call with a per-event
loop wrapped in try/catch. On exception: log event type, messageId,
timestamp, peer origin, and the error message; continue to the next
event.
Previously, a single poison-pill event (e.g., UNIQUE conflict from
a malformed relay payload) would throw, be caught by the outer
try/catch, and block lastSyncedAt from advancing — causing every
future activation to retry the same broken window indefinitely.
The final 'replayed N events' log line now reports '(K skipped due
to errors)' when K > 0, surfacing the count to operators. Individual
event failures are logged via console.error with enough context to
debug or replay manually.
Trade-off documented in docs/systems/federation.md: forward progress
of the sync pipeline takes priority over strict at-least-once
delivery. An event that fails to process is lost to the receiver
unless replayed manually.
Discovered during live verification of #10b Scenario 3: the switch
in ensurePeered had no case for needs_attention, so it fell through
to performHandshake. Because the /peer/accept idempotent-200-no-update
safeguard covers needs_attention on the inbound side, the remote
returned 200 without writing the new secret, and performHandshake
transitioned the local peer to 'active' on the 200 response — auto-
healing a state that requires admin intervention.
Affected paths: sendCallRelay non-blocking warm-up (used by typing
relay); any future caller of ensurePeered on a needs_attention peer.
Not affected: resolvePendingPeers (already filters on status='pending').
Fix: explicit case 'needs_attention' returning { status: 'rejected',
error }. Caller observes the rejection and does not advance state.
Four event types previously bypassed appendMutationLog, making
them unrecoverable via /api/federation/sync after peer inactivity:
- queueDmCloseRelay (dm_close, dm_reopen)
- queueReadStateRelay (read_state_update)
- handleSizeRejection in federationWorker (file_rejected)
- profile PATCH route (profile_update) — two call sites,
one appendMutationLog per profile change (not per target origin)
The /api/federation/sync response builder is extended to
serialize these event types in the next task.
Every code location that sets federation_peers.status='active'
now invokes onPeerActivated(peerId, reason). HTTP handler sites
use fire-and-forget (.catch(log)) so the response isn't blocked
by sync-pull pagination. The worker-internal health-check site
awaits the handler since the tick is already async.
Sites: /peer/initiate, /peer/accept (4 branches), /approval-
requests/:id/approve, health check recovery, ensurePeered/
performHandshake.
Replaces the silent UNIQUE-swallow placeholder branch. Each peer
status has an explicit branch:
active/pending/unreachable: race-catch — re-fetch peer row and
enqueue via matchedPeers. Previously skipped silently, losing
real-time delivery under asymmetric failure.
awaiting_approval/needs_attention/rejected/revoked: drop with
logged reason. Mutation log still captures; sync-pull on
activation replays.
default: exhaustiveness check (no 'as never' cast) — TypeScript
enforces that every status value is handled explicitly.
Missed in 02a1ed7. The new sync-pull path in federationPeerActivation.ts
uses a dynamic import of processRelayEvents from routes/federation.js;
the static import in federationWorker.ts is no longer used after
runInitialSyncForNewPeers deletion.
The per-peer sync body is now syncPeerMutationLog (in the new
peer-activation module), invoked via onPeerActivated. The startup
path scans for status='active' AND lastSyncedAt=0 and calls the
unified handler for each — same trigger condition as before, unified
code path with runtime transitions.
Two-invariant handler: resetOutboxBackoff + syncPeerMutationLog.
In-flight map keyed by peerId coalesces concurrent activations —
a second call for a peer whose activation is still running shares
the same promise. Errors are swallowed and logged — the handler
never throws so fire-and-forget callers at HTTP handler sites
are safe.
- Add pagination-advance test: verifies since=checkpoint on second
iteration within a pass, and each pass re-seeds since from
peer.lastSyncedAt (not carried from prior pass).
- Eliminate four peer! non-null assertions by capturing the narrowed
value in activePeer after the guard.
- Tighten bodyObj type from Record<string, unknown> to a local
SyncRequestBody type alias.
- Drop the no-op federationRelayEnabled UPDATE in test beforeEach
(default is already 1 per baseline migration).
Three-pass pull-sync (dm, friend, profile) from peer's
/api/federation/sync endpoint, paginated. Seeds sinceTimestamp from
peer.lastSyncedAt so a recovered peer pulls only the delta. Updates
lastSyncedAt to Date.now() on full success; leaves it untouched on
transient failure so the next activation retries the same window.
Replaces the body of the soon-to-be-removed runInitialSyncForNewPeers.
Unconditionally resets nextRetryAt=now and attempts=0 for all
outbox entries of the given peer. No WHERE filter on nextRetryAt —
resetting attempts=0 on already-eligible rows is the correctness fix:
without it, a previously-failed entry keeps stale attempts, and its
next failure uses BACKOFF_SCHEDULE_MS[attempts] (5min to 24h) on a
peer that just recovered.
Empty stubs for onPeerActivated, resetOutboxBackoff, syncPeerMutationLog,
and startupBootstrapSync. Functions are filled in by subsequent tasks
following TDD cycles.
HEALTH_CHECK_INTERVAL_MS was 1 h, but ROTATION_GRACE_PERIOD_MS is 15 min.
Phase skew between two peers' health-check ticks could stretch rotation
finalization desync up to ~1 h, during which signatures from the already-
finalized side verify against the other side's primary-only secret (grace
has expired; verifyPeerSignature stops trying the pending secret). With
AUTH_FAILURE_THRESHOLD = 5 and the existing backoff schedule, this
occasionally tripped legitimate rotations into needs_attention.
Setting the interval to 15 min (= ROTATION_GRACE_PERIOD_MS) guarantees a
finalization tick fires within one grace window on each side, so the
cross-verification window where one peer signs with NEW while the other
still treats NEW as pending cannot outlast the grace period.
Per-tick cost is negligible for the worker's steady state: the only
network fetches are per-active-peer /peer/rotate calls when the 90-day
rotation interval hits (rare) and per-unreachable-peer /instance/info
health pings (bounded by outage count). Going lower than 15 min would
reduce the residual desync but increase tick overhead with diminishing
returns; 15 min is the grace-period-aligned value that the original spec
("runs hourly") deviated from without justification.
Follow-up #20 from S2S DM unification backlog; reduces #19 false-positive
rate (outbox auth-failure transition) on legitimate rotations.
Pairs with the new 401/403 handler — a 2xx relay confirms HMAC trust is
healthy so the counter should clear. Mirrors the existing
consecutive_failures reset for network-layer health.
Code review of the previous commit found that the backoff branch of the
new 401/403 handler delegated to handleOutboxDeliveryFailure, which
double-dips by also incrementing consecutive_failures (the network-layer
counter that drives the 'unreachable' transition at threshold 10). Per the
design spec §State Machine Changes → Reset logic, auth failures must
increment consecutive_auth_failures ONLY.
Split handleOutboxDeliveryFailure into:
- applyOutboxEntryBackoff: just the per-entry backoff update (safe to call
from the auth-failure path)
- handleOutboxDeliveryFailure: entry backoff + peer's consecutive_failures
bump (network-error path only)
Also adds a console.warn to the backoff branch so operators can diagnose
clock-skew and rotation-grace incidents before the peer hits the terminal
threshold.
Part of backlog #19.
The previous handler (commit ce33ccf + its 403 extension) wiped hmac_secret
and reset peer status to 'pending' on any 401/403 from an active peer. This
collapsed three distinct failure modes — transient clock skew, legitimate
split-brain, active MITM attempt — into "silently establish new trust
immediately." The remote's /peer/accept idempotent-200-no-update safeguard
then prevented the re-handshake from actually working, producing a 1-req/sec
loop observed during backlog #16 verification.
New behavior: increment consecutive_auth_failures, apply backoff to outbox
entries. At AUTH_FAILURE_THRESHOLD (5) transition to needs_attention,
preserve hmac_secret, surface delivery-impossible to affected users,
notify admins. Secret is NEVER wiped in response to a network-observed
401/403.
Part of backlog #19.
Pure function deciding whether the next 401/403 from an active peer
triggers backoff or a transition to needs_attention. Threshold = 5,
corresponding to ~21.5 min of the existing BACKOFF_SCHEDULE_MS.
sendCallRelay now returns CallRelayResult with a typed reason on failure.
When the peer is not already active (or unreachable), runs a racePeering
against CALL_PEERING_TIMEOUT_MS (3s). Background handshake is not aborted
on race loss — next attempt succeeds.
sendTypingRelay passes peeringTimeoutMs:0 so typing never blocks on a
handshake; instead a warm-up ensurePeered runs in the background for any
non-active peer so the NEXT relay (message, call, or typing) benefits.
Addresses code review on b22a7bd: (1) a rejected handshake now returns
{ status: 'failed', error } instead of throwing, keeping the structured
contract; (2) the "background handshake" warn only fires when the
timeout arm wins — not when the handshake is itself the race winner by
rejection. Timing tests migrated to vi.useFakeTimers for determinism.
Regression test added for the handshake-wins-by-rejection case.
Exports `racePeering(origin, timeoutMs, ensurePeeredFn?)` that races
`ensurePeered` against a deadline. On timeout, the background handshake
continues (warming the peer for the next attempt) and a warn-logged
.catch() prevents unhandledRejection. Injectable `ensurePeeredFn` param
enables full DI in tests without mocking module internals.
Added federation_peers_changed (no-payload signal) broadcast from every
peer state mutation, and federation_approval_request_received when a new
approval request is queued. Client subscribes via onFederationPeersChanged
callback registry. FederationPanel and PendingApprovals debounce-refetch
on any event. sendToAdmins helper broadcasts only to admin users.
autoAcceptPeering means 'don't accept peering initiated by others', not
'don't initiate peering ourselves'. Two checks were incorrectly blocking
outgoing peering when auto-accept was off:
1. ensurePeered() refused to auto-initiate — reverted. When a local user
sends a DM, the server should initiate peering. The remote's
peer/accept decides whether to accept or queue.
2. queueOutboxEvent() refused to create placeholders — reverted. The
outbox needs placeholders to queue entries. Without them, DM relay
silently fails.
1. queueOutboxEvent no longer creates pending peer placeholders when
autoAcceptPeering is disabled — prevents bypassing the admin's
peering control
2. Approval endpoint checks for 202 before response.ok — when the
remote also has autoAcceptPeering off, sets peer to awaiting_approval
instead of incorrectly activating it
3. awaiting_approval status added to Federation panel UI — status label,
colors, filter options so these peers are visible and manageable
When all peers are pending (no active peers with outbox entries),
processOutboxTick returned early at line 141 before reaching
resolvePendingPeers at line 302. Pending peers were never resolved
because the only code path to resolvePendingPeers was after the
active-peer delivery loop — which never ran.
getGroupDmTargetOrigins() returned undefined for 1-on-1 DMs, which
queueOutboxEvent() treated as 'broadcast to all existing peers'. When
no peers existed, nothing was queued and no handshake was ever triggered.
Now always computes target origins from DM participants so the pending
placeholder creation path runs, enabling ensurePeered() → peer/accept
→ approval queue flow.
ensurePeered() now checks the local autoAcceptPeering setting before
initiating new peering. When disabled, only admin-explicit peer/initiate
and approval-request approve bypass this check. Closes the bypass where
client peer/ensure or outbox worker could auto-initiate outward peering
even when the admin intended to control all peering.
Jannis Braun