Manual ownership transfers between two federated instances diverged because
`dm_channels.ownerHomeInstance` was stored as a BARE host (`orbit.ddns.net`)
for federated owners — via `transferGroupDmOwnership` copying `users.homeInstance`
verbatim — while `sourceInstance` always arrives as a full URL on the wire.
`processOwnershipTransferEvent` and `processMemberRemoveEvent` then compared the
two with strict equality and rejected legitimate inbound events as
`unauthorized_source`, keeping ownership permanently divergent across peers.
Live DB inspection on the two test instances confirmed both rows (nova + orbit)
had a BARE `owner_home_instance`, matching the bug report exactly.
Three compounding fixes:
1. Receiver authority checks now compare via `normalizeOriginForCompare` so
legacy bare-vs-full rows accept legitimate transfers (and kicks).
2. New `canonicalizeHomeInstance` helper in `federationAuth.ts`; every write
site that persists `ownerHomeInstance` (`transferGroupDmOwnership`, group DM
creation, lazy federation in member-add, `processMemberAddEvent` bootstrap,
`processOwnershipTransferEvent` receiver storage) routes through it. Full URL
is the canonical storage form, matching how `sourceInstance` arrives.
3. `dm_owner_updated` WS event extended with optional `newOwnerHomeUserId` and
`newOwnerHomeInstance` fields. Client `updateDmOwner` writes them when
present and leaves existing values untouched otherwise (legacy-server safe).
Without this, `getOwnerInstanceForDm` returned the previous owner's home
after a successful WS broadcast, routing the next owner-only op to the wrong
instance.
Coverage: new `federation.ownershipTransfer.test.ts` (7 receiver tests including
the headline bare-vs-full regression and the dedup replay guard); new bare-vs-full
case in `federation.kick.test.ts`; two new client-side cases in
`groupDm.ownerRouting.test.ts` covering both the extended-payload write path and
the legacy-server passthrough. Tests: 1053 server + 364 web, all green.
Specs updated: `dm-system.md` historical bugs + frontend handler table + WS
state-change events table; `federation.md` `ownership_transfer` receiver flow;
`websocket.md` event-fields table.
getProfileReferencedFilenames() didn't include dm_channels.icon, so the
storage janitor deleted group DM icons within ~1 hour:
- Owner instance: PATCH /api/dm/:id leaves an attachments row with
messageId=null and dmMessageId=null. After 1h, getUnlinkedAttachments
flags it and cleanupStorage phase 2 deletes the file because the
filename isn't in profileReferenced.
- Receiver instance: downloadProfileAsset writes the icon directly to
uploadDir with no attachments row. cleanupStorage phase 1 treats it
as orphaned and deletes it.
Fix: include dm_channels.icon (non-null, not soft-deleted, not http://)
in the profile-referenced set. Soft-deleted DMs are excluded so their
files still get reaped by cleanupSoftDeletedDmChannels. Absolute URLs
are skipped because they live on a remote instance.
Also mirror the avatar precedent at the PATCH endpoint by deleting the
new icon's standalone attachment row — the file is now protected via
dm_channels.icon, matching users.ts:473.
Two follow-on bugs from the initial S2S presence rollout:
(1) New friend stuck offline until they reload: presence_update fires only on
transitions, so a remote user already online when their stub is created
locally never receives a relay event seeding their actual status. The
stub defaulted to 'offline' at creation and stayed there until the next
transition. Fix: extend FederationRelayProfileSnapshot +
FederationUserLookupProfile with status. Sender-side buildProfileSnapshot,
getDmParticipants, and lookup endpoint responses populate it for native
users only (replicated stubs hold stale status owned elsewhere).
resolveOrCreateReplicatedUser uses hints.status to seed the new row's
status column. Threaded through every call site (DM participants, group
bootstrap, friend events, ownership transfer). Stub backfill worker also
heals existing rows whose status was stuck at 'offline' from creation.
(2) 'Online' text updates but green avatar dot stays grey on the same page:
spaceStore.updateMemberPresence patches members[] (which feeds space UIs)
but never patches userViews — the cache useCanonicalUserView reads from.
The Avatar in FriendItem reads canonical.status; the text reads
friend.status (socialStore). Two sources, one stale until full
user_updated arrives. Fix: updateMemberPresence now mirrors status into
matching userViews entries, so canonical-view consumers re-render with
fresh status the moment the WS event lands.
The outbox worker rebuilds FederationRelayEvent objects from stored JSON via
an allowlist of known fields. presenceUpdate was missed when presence_update
events shipped, so peers received events with eventType='presence_update' but
no payload — rejected with missing_presence_update_payload on every tick.
onPeerActivated re-emits a presence_update for every relationship-related
online native to the activating peer (relationship = friend with peer-stub /
DM-mate with peer-stub / replicatedInstances opt-in for peer origin). Scope
bounded by relationship count, not native count — flap recovery cost stays
proportional to actual interaction surface.
onPeerDeactivated flips every replicated stub from that peer to offline and
broadcasts a local presence_update so connected friends/DM-mates/space-co-members
see them disappear immediately, instead of seeing stale online until next signal.
Re-snapshot on every activation (incl. health-check unreachable→active flap)
is load-bearing for correctness — markPeerStubsOffline ran on the prior
deactivation and presence is not in the mutation log.
New FederationPresenceUpdatePayload + queuePresenceRelay() helper. Five WS
sites now project the native user's status (and optional activities) to all
active peers via the outbox: WS auth-success, finalizeDisconnect,
manual presence_update, activity_update, showActivity-toggle clear.
Outbox-only (no mutation-log entry) — presence is ephemeral; the upcoming
peer-activation hook re-emits a fresh snapshot so peers recovering from
unreachable converge without history replay. No-op for replicated users.
Heals existing legacy stubs (created when resolveOrCreateReplicatedUser used
homeUserId@domain) by asking the peer for the canonical username via
lookupRemoteUserByHomeId and rewriting the local row. Idempotent and
collision-safe.
Wired into onPeerActivated (per-origin) so future peer flaps re-attempt for
stubs whose home was unreachable on a prior pass, and into a one-shot pass at
startupBootstrapSync for all currently-active peers (not just first-time
lastSyncedAt=0 peers).
HMAC-authenticated, rate-limited (60/min/peer) endpoint that resolves a
homeUserId on this instance to its canonical username + profile snapshot.
Native non-deleted users only. Mirrors /users/lookup's auth shape.
Adds lookupRemoteUserByHomeId to federationLookup.ts as the client-side
helper. Used by the upcoming stub-backfill worker on peers that hold legacy
snowflake-named replicas of users now visible by their real handle.
Adds an explicit override for the federation transport URL returned by
getOurOrigin(). When unset, behaviour is unchanged (https://${DOMAIN} ->
http://localhost:${PORT} dev fallback). Intended for reverse-proxy /
dev-without-TLS deployments where the public origin must be advertised
explicitly (typically http://...) and differs from the bare DOMAIN
value used for federated identity.
Wired via config.publicOrigin (envOptional('PUBLIC_ORIGIN')) so the
override flows through the existing config layer rather than scattering
process.env reads. Trailing slash is stripped for symmetry with
peer.origin storage.
docs/systems/federation.md gets a "Public Origin Override" subsection
under §14 Background Workers documenting the resolution order.
hydrateReplicatedUserProfile now calls downloadProfileAsset and stores bare local filenames, falling back to absolute URLs only on download failure. It also fills empty fields only — no longer clobbering local files written by processProfileUpdateEvent. Adds an idempotent startup backfill that converts existing http-prefixed avatar/banner rows on replicated users into local files, so federated profile pictures keep rendering when the home instance is offline.
Adds an admin-driven sweep on top of the existing 24h auto-expire so
operators can see and reap abandoned `.tus/` sessions without waiting.
- storageJanitor: extract `walkTusDir(predicate)` helper, add
`getStaleTusInfo` + `cleanupStaleTusSessions(thresholdMs, dryRun)`;
refactor `cleanupTusStragglers` to delegate while preserving its
janitor-tick `{ removed }` contract.
- StorageStats gains `staleTusSessions` + `staleTusSize` (fixed 1h
display threshold).
- New `POST /api/admin/storage/cleanup-tus` route with
`maxAgeHours` validation (positive finite number, default 1) and
`dryRun` support; admin-gated.
- StoragePanel: 6th overview card "Stale Uploads" + new cleanup
subsection mirroring the media-cleanup pattern (preview-then-clean
with shared result panel styling).
- Tests: 8 new janitor tests covering empty dir, threshold filtering,
dry-run vs live, oldest-mtime tracking, subdir skipping, and the
override path on the existing straggler sweep. New
`routes/admin.test.ts` covers auth/admin gates, validation (zero,
negative, NaN), default `maxAgeHours`, dry-run vs live unlink.
- Docs: `uploads.md` §Janitor expanded to the full lifecycle (cancel
DELETE, discard DELETE, auto-expire, straggler sweep, admin route);
`admin.md` Storage Management updated with the new endpoint and
StorageStats fields.
POST /api/dm/space-invite was hanging 5s and returning invite_invalid
for any local-space invite. fetchSpaceInviteSnapshot was being called
against our own public domain from inside the backspace container, which
fails (Docker NAT loopback) and aborts on timeout.
Add getLocalInviteSnapshot — reads the snapshot directly from the DB —
and branch in dm.ts so local invites bypass the HTTP roundtrip entirely.
Cross-instance invites still go through fetchSpaceInviteSnapshot with
its existing SSRF guard.
Also refactor the GET /api/spaces/invite/:code/preview handler to use
the same helper, keeping the snapshot shape in one place.
Tests assert fetchSpaceInviteSnapshot is NOT called for the local case
(critical regression guard) and that the cross-instance path still hits
the HTTP fetch.
Quality-review polish on Task 8:
1. InviteUnavailableError gains a typed public readonly `reason` field
(the union 'not found' | 'revoked' | 'expired' | 'exhausted'). Task 11
route handler can switch on the discriminant to produce user-facing
copy without parsing the message string.
2. The original "aborts transaction if insertUser throws" test was
vacuous — the callback threw before any DB write, so SQLite ROLLBACK
never fired and the post-conditions were trivially true. Replaced
with two tests: one that explicitly validates the synchronous
short-circuit (no DB work happens at all), and a second that writes
a real users row inside the callback then throws AFTER the write,
proving the SQLite ROLLBACK actually reverts the in-callback write.
Quality-review polish: a one-line comment over the empty-updates guard
in reinstateInvite explains why removing it would re-leak a confusing
Drizzle error. A new test verifies that when Path A (revoked->active)
fails its post-state check, the original token is preserved by the
SQLite transaction rollback (not replaced by the would-be new token).
Inside patchInvite/revokeInvite txn bodies, all reads now go through
the tx proxy. Pre-Task-7 hygiene: locks in the consistent pattern that
reinstateInvite (Task 7) and redeemInvite (Task 8) will copy.
Createinvite test failures now pin to InviteValidationError, catching
regressions where the wrong error class would otherwise pass silently.
Both functions wrap their read-modify-write in a Drizzle better-sqlite3
db.transaction((tx) => ...) with in-txn re-fetch so concurrent admin
mutations are serialized by SQLite's writer lock.
- patchInvite: 404 on missing, 409 on revoked, 400 on maxUses < usedCount,
allows expiresAt to be moved into the past (effective soft-shut).
- revokeInvite: 404 on missing, 409 on already-revoked (explicit reject,
not silent no-op).
Also extracts foldUsername() to collapse the duplicated
(username, isDeleted) -> display string fold across resolveCreatorUsername,
listInvites, and listRedemptions (deferred refactor from Task 5 review).
Note: the plan's example used db.transaction(cb)() with an IIFE,
which is the raw better-sqlite3 signature. Drizzle's wrapper
returns the callback's return value directly, so we use the
(tx) => ... form consistent with the rest of the codebase
(userDeletion, federation, channels, etc.).
Tests: 36 invite-service tests pass (27 prior + 9 new).
Full server suite: 40 files / 311 tests pass.
Adds two query helpers to inviteService:
- listInvites(filter): single-query LEFT JOIN against users to surface
createdByUsername, with status filtered in TS via the canonical
inviteStatus() derivation. Avoids N+1 the spec calls out (§3.1).
'archived' = expired | exhausted | revoked. Sort: createdAt DESC.
- listRedemptions(inviteId): LEFT JOIN against users via userId to
expose currentUsername alongside the registrantUsername snapshot.
Three null-handling branches per spec §3.1: live (username),
tombstoned ('Deleted User', isDeleted=true), and hard-deleted
(userId null, currentUsername null, isDeleted false).
Sort: redeemedAt DESC.
Also fixes a mistitled DB-miss test in getInviteByToken: the original
'returns null when token not found' used a 24-char string that fails
the format regex *before* the DB lookup. Split into two tests covering
both the format-reject path and the well-formed-but-missing path.
40 files / 302 tests passing.
users.status was only flipped back to offline by the WebSocket disconnect
path (5s grace timer in ConnectionManager). Process exits (deploy/crash/OOM)
lose those in-memory timers, freezing any non-offline row at its last value
and making the user appear permanently online to friends and space co-members.
Confirmed in production on the Pi instance: a user appeared online for ~3
days with no live socket.
Add resetStalePresenceOnBoot() in utils/presenceBoot.ts and call it from
index.ts after getDb()/seedDatabase() and before WebSocket route registration.
The reset is federation-safe: it only updates rows where home_instance IS
NULL (replicated stubs are projections of remote presence and must not be
stomped) and is_deleted = 0 (tombstoned users are excluded from broadcasts).
Also remove the redundant status='online' write from POST /api/auth/login.
A successful REST login does not imply a live socket; the WS auth handshake
is the single source of truth. Login alone could otherwise produce the same
stuck-online row when a client logs in and never establishes a WS.
Tests cover: locally-homed online/idle/dnd reset, replicated rows untouched,
tombstoned rows untouched, idempotence, mixed populations.
Updates docs/systems/activity-presence.md (Connect/Disconnect Flow, new Boot
Reset section) and docs/systems/auth.md (login no longer mutates status).
Two correctness/defense fixes plus regression tests in the existing
in-memory drizzle test file.
1. Reverse-direction idempotency. The sender-side path in social.ts
checks BOTH directions of friend_requests and returns 409
incoming_request_exists when an opposite-direction row exists. The
receiver only matched from->to, so cross-fire (alice@A and bob@B both
click "add friend" near-simultaneously) produced two opposite
pending rows on each instance. The receiver now silent-accepts when
either direction matches a pending row, mirroring the sender's
both-direction check.
2. Self-target guard (defense-in-depth). Reject events whose
from-identity equals to-identity (after normalizeOriginForCompare)
with a new receiver-acknowledged 4xx code self_target_invalid.
Sender's local cannot_friend_self should catch this, but the
receiver does not trust upstream validation. Added to
TERMINAL_REJECTION_REASONS so the standard rollback fires
(mapped client-side to peer_rejected). Logged at console.warn.
Spec updates: social.md inbound contract now documents both-direction
idempotency and the self-target guard; federation.md and the
s2s-friend-add design spec list the new terminal rejection reason.
The third describe block (`ensurePeered needs_attention handling`) was
the last federation test still using the legacy `vi.doMock` +
hand-crafted drizzle query-chain pattern. Convert it to the converged
in-memory better-sqlite3 + real drizzle pattern used by every other
federationPeering.* test (exemplar:
federationPeering.approvalToken.test.ts).
The first two describe blocks (`EnsurePeeredResult type`, `racePeering`)
did not use vi.doMock and remain byte-identical. The needs_attention
test's behavior is preserved 1:1: same `it(...)` description, same
assertions (status === 'rejected', error contains 'needs_attention',
fetch never called).
Module-level `vi.mock` calls now apply to the whole file, but the
type-only and racePeering tests don't exercise the mocked modules, so
there is no behavioral interaction.
No production code changes. No new test scenarios. No skipped/only/
commented blocks. All 352 server tests pass.
Live verification caught a stale-UI bug: when admin denial / approval
fanout / janitor expiry cascaded subscriber rows away, only the
peering_notification_received WS event fired (which only refetches
the notification list). The user's pending-subscriptions section
stayed stale until manual refresh.
Each terminal-state path now also fires peering_subscription_changed
to affected users so their pending list refreshes alongside the new
notification. Fixed in:
- onPeerActivated.fanoutOutboundSubscribers (approval path)
- handleOutboundDeny (admin denial)
- cleanupExpiredApprovalRequests (janitor expiry; also adds the
notification-received broadcast that was previously deferred to
next-page-load only)
Task 9's first pass replaced inbound expiry's signed /peer/denied
POST with a plain delete, citing symmetric independent expiry as
the design. The spec at §4.10 explicitly said 'inbound row cleanup
unchanged' — that was scope creep, not a fix. Inbound expiry now
preserves the pre-branch behavior verbatim: signed POST to remote,
delete only on success, retry on failure. Outbound expiry's fanout
logic (the actual Task 9 scope) is unchanged.
- GET /api/federation/peering-notifications (?unread=1 filter)
- POST /api/federation/peering-notifications/:id/read (per-row mark-read)
- POST /api/federation/peering-notifications/read-all (bulk mark-read,
preserves already-read readAt; returns affected count)
- janitor: outbound expired rows fan out kind='expired' notifications
to each subscriber before cascade-deleting parent (replaces Task 1's
scaffolding 'continue' guard); inbound expired rows are deleted outright
(both sides expire independently — no cross-instance network call)
- janitor: 30-day cleanup pass for read notifications
(cleanupReadPeeringNotifications); unread rows persist indefinitely
- cleanupExpiredApprovalRequests is now sync (no async network IO)
Notes:
- No WS broadcast on janitor expiry — offline users see notifications on
next GET, matching the persistence guarantee of the notifications table.
- The previous /peer/denied network call on inbound expiry is removed;
symmetric per-side expiry now handles termination on both sides.
Tests: 12 new (3 + 4 endpoint cases on 3 endpoints; 2 outbound/inbound
janitor expiry cases + 1 not-yet-expired guard + 1 retention sweep);
247 server tests passing total.
The fanoutOutboundSubscribers helper is called from onPeerActivated,
the single point all peer activations flow through (queue approval,
/peer/initiate, autoAccept=1 remote, mutual-approval token verification).
Subscribers see kind='approved' notification, parent + subscriber rows
cascade-delete. Critical: cleanup triggers on status->active, not on
the local approve action — when the remote also gates, the peer goes
to awaiting_approval first and subscribers must remain queued.
- social.ts friend-add: user_action, with 409 peer_pending_local_admin
when gate fires
- /peer/ensure: user_action, surfaces peeringStatus: 'admin_required'
- sendCallRelay (typing warm-up + call relay): system intent
- federationWorker resolvePendingPeers: system intent (defensive — gate
is unreachable from here since pending rows already exist)
- CallRelayFailureReason: peer_admin_required added (mapped to
peer_transient_failure on the user-facing event surface, since system
intent should never legitimately surface admin_required)
- Test files: thread intent arg through racePeering and ensurePeered
calls (positional shift from racePeering signature change)
- outboundGate.test.ts: tighten noUncheckedIndexedAccess access via
non-null assertions after toHaveLength()
- docs/systems/social.md: peer_pending_local_admin error code documented
The pre-existing trust-guard's query filtered peer_approval_requests by
origin only. After Task 3 added outbound rows to the same table, the
guard started matching the user's own queued outbound row on retry,
returning 'rejected' with a misleading 'admin must resolve pending
approval' copy instead of the intended 'admin_required'. The variable
name (pendingInbound) and comment block already described the intent
as inbound-only — the query just didn't match. Adding direction='inbound'
to the where clause restores the intended behavior.
- New 'admin_required' EnsurePeeredResult variant.
- Required intent argument (no optional default) prevents future
user-initiated callers from silently getting system behavior.
- Gate runs only when no peer row exists; toggling autoAccept later
does not retroactively gate established peer rows.
- queueOutboundApproval helper upserts parent + subscriber rows and
broadcasts to admins + user.
- Schema enum-narrows direction ('inbound' | 'outbound') and
notifications.kind ('approved' | 'denied' | 'expired') at the column
level; drizzle generate confirmed no migration delta.
- Existing call sites now fail typecheck — fixed in Task 5.
Receiver-side defense — closes the trust-bypass class the cheap fix
(4533e36) cannot cover. The /peer/accept handler's awaiting_approval
branch now requires an approvalToken matching the one stored on the
local peer row before promoting to active. Without a match:
- autoAccept=0 falls through to queueApprovalRequest (no bypass; new
approval-request queued, existing awaiting_approval row untouched).
- autoAccept=1 falls back to permissive promotion (no regression vs
prior behavior, since autoAccept=1 would accept any inbound regardless).
Successful match also deletes any stale approval-request row for the
origin to prevent debris accumulation from prior bypass attempts.
Refactor: queueing path extracted to a top-level queueApprovalRequest()
helper so both the no-existing-peer case and the awaiting_approval
mismatch fallback share one implementation. Helper generates a fresh
single-use token on every call.
Adds 'accept_awaiting_approval_fallback' variant to PeerActivationReason
to distinguish the autoAccept=1 fallback path from the verified path
in onPeerActivated audit logs.
Spec §3.5, §3.6.
Test counts: 289 → 301 (+12).
- 202 response: parse approvalToken from body and store on the local
federation_peers row alongside status='awaiting_approval'. Legacy
receivers that omit the field result in null stored token, handled
gracefully by the verification logic landing in subsequent commits.
- 200 response: clear approvalToken in the same UPDATE that sets
status='active' (single-use consumption per spec §3.2).
Test coverage: 4 tests in federationPeering.approvalToken.test.ts covering
JSON token, missing token (legacy), non-JSON body, and 200 clear after
prior token storage.
Jannis Braun