Six S2S-HMAC endpoints repeated the same inbound-auth preamble verbatim
(parse federation headers -> resolve active peer -> optional per-peer rate
limit -> verify HMAC signature -> nonce replay protection). Extract it into
authenticateS2SPeer() so the trust boundary has a single, tested definition.
Adopters (preamble only; every post-auth side effect, body validation, and
response is unchanged):
- DELETE /api/federation/identity (no rate limiter; warns on missing nonce)
- POST /api/federation/relay (relay limiter; warns; keeps in-handler
epoch-baseline populate + nonce ratchet)
- POST /api/federation/sync (no limiter; warns with the [sync] tag;
keeps in-handler nonce ratchet)
- POST /api/federation/users/lookup (lookup limiter, Retry-After 60)
- POST /api/federation/users/by-home-id (same)
- POST /api/federation/verify-attach-proof(shares lookup bucket, Retry-After 60)
Deliberate non-adopters, each keeping a load-bearing gate the helper would
flatten (documented at each site + in the helper docstring):
- POST /api/federation/epoch gates status != 'revoked' (peer recovery),
400 on missing headers, no nonce check
- POST /api/federation/peer/rotate active-only but no nonce check
- POST /api/federation/peer/denied awaiting_approval gate (404/409), synthetic
no-grace secret verify
Behavior-preserving. The rate limiter is injected (plain { limited, retryAfter }),
so the limit still fires BEFORE signature verification. The only ordering change:
/relay's opportunistic epoch-baseline populate now runs just after the shared
preamble (i.e. after the nonce check) instead of between signature and nonce.
This is provably equivalent for every reachable honest-peer state (a duplicate
nonce means the baseline is already non-null; a valid-signature-but-no-nonce
request from a nonce-supporting peer is unreachable in transit and carries no
security/correctness consequence) and the populate is documented as not
affecting relay accept/reject.
Adds a dedicated unit test covering the full decision table (headers, peer
status, rate-limit + Retry-After, rate-limit-before-signature ordering,
signature, nonce duplicate/missing, log flag + context suffix, success). Full
server suite green (804 tests).
routes/federation.ts had grown to 7.6k lines, spanning HTTP route
registration, federated identity resolution, ~30 inbound relay event
processors, DM reconciliation, and rate-limiting internals — too large
to review or hold in context, and awkward to change safely.
Split the implementation into 18 focused modules under routes/federation/
(helpers, events/, handlers/) and keep routes/federation.ts as a thin
barrel that re-exports the public API and composes the HTTP registrars
into federationRoutes(). No import paths change anywhere else.
Pure move, no behavior change:
- 61/61 named functions byte-identical; only deltas are 2 dynamic-import
paths adjusted for the new directory depth
- public export surface unchanged (barrel re-exports all 22 symbols)
- all 30 endpoints preserved (identical verb+path set)
- typecheck, build, and full server suite (790 tests) green
Docs: update federation.md source-file map; add split design doc.
A reset peer reaches needs_attention via the auth-failure path (HMAC desynced by
the new incarnation) without passing through unreachable, so the 5s recovery
probe never saw it — detection waited up to a full 15-min health-check cycle
before 'Re-peer & heal' surfaced. Extract detectResetForPeer() and fire it
event-driven at the transition, plus a startup sweep for already-stuck peers.
15-min tick remains the backstop.
A reset peer can reach needs_attention via the auth-failure path (HTTP up,
401/403 from a new incarnation crossing AUTH_FAILURE_THRESHOLD) without ever
passing through unreachable, so the unreachable-only recovery probe never
observes its epoch change and no reset journal is created — leaving a later
manual Re-peer with nothing to heal.
Add detectResetOnNeedsAttentionPeers() to the 15-minute health-check tick:
probe needs_attention peers with a non-null baseline (excluding those already
peer_reset_detected) and call markPeerReset on an observed epoch mismatch.
Detection only — never recovers a needs_attention peer to active; baseline
(peer_instance_id) and hmac_secret untouched.
Add healResetIncarnation (federationReset.ts): fires from onPeerActivated after
an authenticated re-peer to soft-tombstone the flagged pure S2S stubs of a reset
peer's dead incarnation, clearing stale friendships/DMs so the reported bug is
fixed. Two mandatory guards: a reason gate (allow-list of 8 genuine handshake
activation reasons; excludes health_check_recovery + startup_bootstrap so their
stale baseline can never silently resolve a journal without healing) and an
epoch comparison (dead_epoch === newEpoch => false alarm, no tombstone). Uses
tombstoneUser(uid, { purgeContent: false }); real federated accounts are left
flagged + intact for Phase 2. Runs outside any transaction. Wire into
onPeerActivated before the mutation-log re-sync.
macOS screen recordings are HEVC inside a .mov container, which Chromium,
Firefox and stock Electron can't decode. The file uploaded fine and a
server-side ffmpeg poster was generated, but inline <video> playback failed
silently — stuck at 0:00 with no error, since AttachmentRenderer had no error
handling. Root cause: the system had no concept of web-playability.
Server detects, client degrades:
- mediaPlayable.ts: classifyVideoPlayable(mimetype, codec) — tri-state
(false = known-undecodable e.g. HEVC/ProRes, true = web codec in web
container, null = unknown/optimistic). Never widens `false` beyond codecs
that fail everywhere, so ffmpeg-less instances keep prior behaviour.
- probeMediaMeta now captures the video codec_name; the upload finish hook
stores the verdict in the new attachments.playable column (migration 0007).
- Flag propagated through every serializer: space messages, DMs, WS, and
federation relay (outbound + inbound) — federation-compatible.
- VideoAttachment component: playable===false renders a download card (poster
+ "Can't play here — download" + name/duration/size) with no dead-player
flash; otherwise plays inline with an onError fallback to the same card.
Specs updated: uploads.md, database.md, federation.md.
Manual ownership transfers between two federated instances diverged because
`dm_channels.ownerHomeInstance` was stored as a BARE host (`orbit.ddns.net`)
for federated owners — via `transferGroupDmOwnership` copying `users.homeInstance`
verbatim — while `sourceInstance` always arrives as a full URL on the wire.
`processOwnershipTransferEvent` and `processMemberRemoveEvent` then compared the
two with strict equality and rejected legitimate inbound events as
`unauthorized_source`, keeping ownership permanently divergent across peers.
Live DB inspection on the two test instances confirmed both rows (nova + orbit)
had a BARE `owner_home_instance`, matching the bug report exactly.
Three compounding fixes:
1. Receiver authority checks now compare via `normalizeOriginForCompare` so
legacy bare-vs-full rows accept legitimate transfers (and kicks).
2. New `canonicalizeHomeInstance` helper in `federationAuth.ts`; every write
site that persists `ownerHomeInstance` (`transferGroupDmOwnership`, group DM
creation, lazy federation in member-add, `processMemberAddEvent` bootstrap,
`processOwnershipTransferEvent` receiver storage) routes through it. Full URL
is the canonical storage form, matching how `sourceInstance` arrives.
3. `dm_owner_updated` WS event extended with optional `newOwnerHomeUserId` and
`newOwnerHomeInstance` fields. Client `updateDmOwner` writes them when
present and leaves existing values untouched otherwise (legacy-server safe).
Without this, `getOwnerInstanceForDm` returned the previous owner's home
after a successful WS broadcast, routing the next owner-only op to the wrong
instance.
Coverage: new `federation.ownershipTransfer.test.ts` (7 receiver tests including
the headline bare-vs-full regression and the dedup replay guard); new bare-vs-full
case in `federation.kick.test.ts`; two new client-side cases in
`groupDm.ownerRouting.test.ts` covering both the extended-payload write path and
the legacy-server passthrough. Tests: 1053 server + 364 web, all green.
Specs updated: `dm-system.md` historical bugs + frontend handler table + WS
state-change events table; `federation.md` `ownership_transfer` receiver flow;
`websocket.md` event-fields table.
Overloading NODE_ENV='test' to silently disable rate limiting layered a
second meaning onto an env that already gates the test-only seed-peer
route. A dedicated DISABLE_RATE_LIMITS env (envBool semantics, matching
DISABLE_FEDERATION_WORKERS) makes intent explicit, defaults off in
production, and leaves room for tests that need to assert real rate-limit
behaviour to opt back in by simply not setting the var.
Adds an explicit override for the federation transport URL returned by
getOurOrigin(). When unset, behaviour is unchanged (https://${DOMAIN} ->
http://localhost:${PORT} dev fallback). Intended for reverse-proxy /
dev-without-TLS deployments where the public origin must be advertised
explicitly (typically http://...) and differs from the bare DOMAIN
value used for federated identity.
Wired via config.publicOrigin (envOptional('PUBLIC_ORIGIN')) so the
override flows through the existing config layer rather than scattering
process.env reads. Trailing slash is stripped for symmetry with
peer.origin storage.
docs/systems/federation.md gets a "Public Origin Override" subsection
under §14 Background Workers documenting the resolution order.
hydrateReplicatedUserProfile now calls downloadProfileAsset and stores bare local filenames, falling back to absolute URLs only on download failure. It also fills empty fields only — no longer clobbering local files written by processProfileUpdateEvent. Adds an idempotent startup backfill that converts existing http-prefixed avatar/banner rows on replicated users into local files, so federated profile pictures keep rendering when the home instance is offline.
Two correctness/defense fixes plus regression tests in the existing
in-memory drizzle test file.
1. Reverse-direction idempotency. The sender-side path in social.ts
checks BOTH directions of friend_requests and returns 409
incoming_request_exists when an opposite-direction row exists. The
receiver only matched from->to, so cross-fire (alice@A and bob@B both
click "add friend" near-simultaneously) produced two opposite
pending rows on each instance. The receiver now silent-accepts when
either direction matches a pending row, mirroring the sender's
both-direction check.
2. Self-target guard (defense-in-depth). Reject events whose
from-identity equals to-identity (after normalizeOriginForCompare)
with a new receiver-acknowledged 4xx code self_target_invalid.
Sender's local cannot_friend_self should catch this, but the
receiver does not trust upstream validation. Added to
TERMINAL_REJECTION_REASONS so the standard rollback fires
(mapped client-side to peer_rejected). Logged at console.warn.
Spec updates: social.md inbound contract now documents both-direction
idempotency and the self-target guard; federation.md and the
s2s-friend-add design spec list the new terminal rejection reason.
federation.md: new 'Approval Token Verification' subsection covering
issuance (queue path generates token, returns in 202), storage on
initiator (federation_peers.approval_token), forwarding from /approve,
verification on receiver's awaiting_approval branch, single-use lifecycle,
backward compatibility, and threat-model boundary (sender-side outbound
gating tracked separately).
database.md: approval_token column documented on both federation_peers
and peer_approval_requests with cross-references to federation.md.
api.md: /peer/accept request + 202 response now show optional
approvalToken field with pointer to the federation spec.
Closes the auto-reconnect trust-bypass: any code path calling
ensurePeered(remote) on an instance with autoAcceptPeering=0 could
previously bypass the admin gate by initiating a fresh handshake to the
remote, which the remote then accepted against its existing
awaiting_approval row.
The trigger surfaced was stores/instanceStore.ts:1010 — the silent
.catch(() => {}) auto-reconnect that fires for any user with the
remote in their replicatedInstances (commonly: any admin). Anyone with
that profile reloading their session activated peering on both sides
without any admin approval action.
Surgical fix: ensurePeered now returns rejected when an unresolved
inbound peer_approval_requests row exists for the target origin. The
legitimate admin-approve flow (routes/federation.ts:1089) does not call
ensurePeered; it deletes the approval-request and does its own direct
fetch to /peer/accept, so this check does not block legitimate approvals.
The receiver-side trust assumption at routes/federation.ts:619-645
(awaiting_approval branch in /peer/accept) still has the same flaw
— an adversarial peer that knows the timing could re-handshake at the
right moment to flip the receiver to active. That deeper trust-model
rework is plan-grade work tracked at internal notes
2026-04-26-peer-handshake-trust-model.md.
federation.md — new undeliverable bucket subsection with three-way
classification table, Path A/B semantics, and wire backward-compat note.
voice.md — no_recipient row in failure-surface table.
websocket.md — DmCallUndeliverableReason union updated to include no_recipient.
dm-system.md — cross-reference to voice.md for no_recipient reason.
Duplicate rejection means the peer already has the message (e.g.,
delivered earlier via outbox AND pulled via sync in the same
window). Retrying will fail identically forever until TTL expires.
Before this patch: duplicate-rejected outbox entries were retained
with attempts++ and exponential backoff, creating log noise and
outbox bloat for up to 30 days.
After: duplicate-rejected entityIds join the terminal set alongside
accepted ones and are deleted from the outbox. Logged at info level
('outbox entry removed (terminal)') to distinguish from warn-level
transient-rejection retries.
Other rejection reasons (attribution_mismatch, processing_error,
etc.) stay on the retry path; some may also be terminal but are
deferred until observed accumulating.
Replace the batch-level processRelayEvents call with a per-event
loop wrapped in try/catch. On exception: log event type, messageId,
timestamp, peer origin, and the error message; continue to the next
event.
Previously, a single poison-pill event (e.g., UNIQUE conflict from
a malformed relay payload) would throw, be caught by the outer
try/catch, and block lastSyncedAt from advancing — causing every
future activation to retry the same broken window indefinitely.
The final 'replayed N events' log line now reports '(K skipped due
to errors)' when K > 0, surfacing the count to operators. Individual
event failures are logged via console.error with enough context to
debug or replay manually.
Trade-off documented in docs/systems/federation.md: forward progress
of the sync pipeline takes priority over strict at-least-once
delivery. An event that fails to process is lost to the receiver
unless replayed manually.
Adds 'Peer Activation Recovery' subsection with call-site roster,
peer-state x outbox-enqueue x recovery matrix, mutation log
coverage table, /api/federation/sync contextType filter values,
and a Known Issues note about the poison-pill edge case.
Also updates stale references to runInitialSyncForNewPeers (removed
in commit 02a1ed7) to point at the unified startupBootstrapSync
path.
/peer/initiate checked `response.ok` to decide whether to activate the
local peer. `response.ok` is true for the full 2xx range, so a remote
that returned 202 (queued for admin approval — autoAcceptPeering off
on their side) caused the local peer to flip to `active` while the
remote had us `awaiting_approval`. The split only self-healed when
the remote admin approved and pushed us an `awaiting_approval → active`
override via the peer_approval_requests inbound path.
The auto-peer flow in federationPeering.ts:performHandshake already
had the correct 202 branch: set local status to awaiting_approval,
broadcast federation_peers_changed, surface a pending outcome. Mirror
it here:
- Check response.status === 202 BEFORE the !response.ok branch so the
fall-through can't reach the activation code.
- Transition local peer to awaiting_approval (not active).
- Broadcast federation_peers_changed so other admin tabs refresh.
- Return 202 with the sanitized peer so the client observes the
queued state distinctly from both success and failure.
Also added the missing federation_peers_changed broadcast on the
activation (200) path for parity with every other peer-state-change
site in the codebase — it was a pre-existing drift that would leave
sibling admin tabs stale after an initiate. Pattern-aligned with
federationPeering.ts:160 and the rest of routes/federation.ts.
Docs: expanded Phase 1 bullets in docs/systems/federation.md to cover
the 200 / 202 / other non-2xx / network-error branches explicitly and
reference the mirrored auto-peer branch.
Verified: pnpm -r typecheck clean (shared + server), vitest 70/70
pass.
Closes#21 from S2S DM unification backlog.
HEALTH_CHECK_INTERVAL_MS was 1 h, but ROTATION_GRACE_PERIOD_MS is 15 min.
Phase skew between two peers' health-check ticks could stretch rotation
finalization desync up to ~1 h, during which signatures from the already-
finalized side verify against the other side's primary-only secret (grace
has expired; verifyPeerSignature stops trying the pending secret). With
AUTH_FAILURE_THRESHOLD = 5 and the existing backoff schedule, this
occasionally tripped legitimate rotations into needs_attention.
Setting the interval to 15 min (= ROTATION_GRACE_PERIOD_MS) guarantees a
finalization tick fires within one grace window on each side, so the
cross-verification window where one peer signs with NEW while the other
still treats NEW as pending cannot outlast the grace period.
Per-tick cost is negligible for the worker's steady state: the only
network fetches are per-active-peer /peer/rotate calls when the 90-day
rotation interval hits (rare) and per-unreachable-peer /instance/info
health pings (bounded by outage count). Going lower than 15 min would
reduce the residual desync but increase tick overhead with diminishing
returns; 15 min is the grace-period-aligned value that the original spec
("runs hourly") deviated from without justification.
Follow-up #20 from S2S DM unification backlog; reduces #19 false-positive
rate (outbox auth-failure transition) on legitimate rotations.
TheZwiss