A reset peer reaches needs_attention via the auth-failure path (HMAC desynced by
the new incarnation) without passing through unreachable, so the 5s recovery
probe never saw it — detection waited up to a full 15-min health-check cycle
before 'Re-peer & heal' surfaced. Extract detectResetForPeer() and fire it
event-driven at the transition, plus a startup sweep for already-stuck peers.
15-min tick remains the backstop.
Also add the positive companion assertion (folded in from a Task 3 review
Minor): findFederatedUser tier-2 STILL returns a NON-detached (orphaned=0)
same-name federated row, locking that the eq(federation_home_orphaned, 0)
clause discriminates on the flag alone and never over-filters legitimate
replicated identities.
All three behaviors pass against the shipped Task 1-3 code; no product-code
change was required.
Finding 1: handleReactionAdd/Remove now drop reactions on a dead 1-on-1
(isDeadOneOnOne) — previously a survivor could react on a Deleted-User
thread and the relay fanned out to all peers via undefined target-origins.
Client Message.tsx withdraws add/toggle reaction affordances for dead DMs
(existing reactions still display read-only).
Finding 2: dmMembership purge test now runs with foreign_keys=ON (matches
prod) and asserts dm_members/dm_messages cascade cleanup on channel purge.
Finding 3: tombstone group-DM ownership transfer filters isDeleted=0 so
ownership can never move to a tombstoned member; covered by a new test.
lookupRemoteUser now maps peer HTTP failures (403/5xx, malformed body) to a
structured {ok:false,reason:'unreachable'} instead of throwing, and the
federated friend-add wraps the call in try/catch as defense-in-depth. A
desynced/unreachable peer no longer surfaces as a raw 500 on a user action.
README.md left unstaged.
A reset peer can reach needs_attention via the auth-failure path (HTTP up,
401/403 from a new incarnation crossing AUTH_FAILURE_THRESHOLD) without ever
passing through unreachable, so the unreachable-only recovery probe never
observes its epoch change and no reset journal is created — leaving a later
manual Re-peer with nothing to heal.
Add detectResetOnNeedsAttentionPeers() to the 15-minute health-check tick:
probe needs_attention peers with a non-null baseline (excluding those already
peer_reset_detected) and call markPeerReset on an observed epoch mismatch.
Detection only — never recovers a needs_attention peer to active; baseline
(peer_instance_id) and hmac_secret untouched.
Add healResetIncarnation (federationReset.ts): fires from onPeerActivated after
an authenticated re-peer to soft-tombstone the flagged pure S2S stubs of a reset
peer's dead incarnation, clearing stale friendships/DMs so the reported bug is
fixed. Two mandatory guards: a reason gate (allow-list of 8 genuine handshake
activation reasons; excludes health_check_recovery + startup_bootstrap so their
stale baseline can never silently resolve a journal without healing) and an
epoch comparison (dead_epoch === newEpoch => false alarm, no tombstone). Uses
tombstoneUser(uid, { purgeContent: false }); real federated accounts are left
flagged + intact for Phase 2. Runs outside any transaction. Wire into
onPeerActivated before the mutation-log re-sync.
macOS screen recordings are HEVC inside a .mov container, which Chromium,
Firefox and stock Electron can't decode. The file uploaded fine and a
server-side ffmpeg poster was generated, but inline <video> playback failed
silently — stuck at 0:00 with no error, since AttachmentRenderer had no error
handling. Root cause: the system had no concept of web-playability.
Server detects, client degrades:
- mediaPlayable.ts: classifyVideoPlayable(mimetype, codec) — tri-state
(false = known-undecodable e.g. HEVC/ProRes, true = web codec in web
container, null = unknown/optimistic). Never widens `false` beyond codecs
that fail everywhere, so ffmpeg-less instances keep prior behaviour.
- probeMediaMeta now captures the video codec_name; the upload finish hook
stores the verdict in the new attachments.playable column (migration 0007).
- Flag propagated through every serializer: space messages, DMs, WS, and
federation relay (outbound + inbound) — federation-compatible.
- VideoAttachment component: playable===false renders a download card (poster
+ "Can't play here — download" + name/duration/size) with no dead-player
flash; otherwise plays inline with an onError fallback to the same card.
Specs updated: uploads.md, database.md, federation.md.
Manual ownership transfers between two federated instances diverged because
`dm_channels.ownerHomeInstance` was stored as a BARE host (`orbit.ddns.net`)
for federated owners — via `transferGroupDmOwnership` copying `users.homeInstance`
verbatim — while `sourceInstance` always arrives as a full URL on the wire.
`processOwnershipTransferEvent` and `processMemberRemoveEvent` then compared the
two with strict equality and rejected legitimate inbound events as
`unauthorized_source`, keeping ownership permanently divergent across peers.
Live DB inspection on the two test instances confirmed both rows (nova + orbit)
had a BARE `owner_home_instance`, matching the bug report exactly.
Three compounding fixes:
1. Receiver authority checks now compare via `normalizeOriginForCompare` so
legacy bare-vs-full rows accept legitimate transfers (and kicks).
2. New `canonicalizeHomeInstance` helper in `federationAuth.ts`; every write
site that persists `ownerHomeInstance` (`transferGroupDmOwnership`, group DM
creation, lazy federation in member-add, `processMemberAddEvent` bootstrap,
`processOwnershipTransferEvent` receiver storage) routes through it. Full URL
is the canonical storage form, matching how `sourceInstance` arrives.
3. `dm_owner_updated` WS event extended with optional `newOwnerHomeUserId` and
`newOwnerHomeInstance` fields. Client `updateDmOwner` writes them when
present and leaves existing values untouched otherwise (legacy-server safe).
Without this, `getOwnerInstanceForDm` returned the previous owner's home
after a successful WS broadcast, routing the next owner-only op to the wrong
instance.
Coverage: new `federation.ownershipTransfer.test.ts` (7 receiver tests including
the headline bare-vs-full regression and the dedup replay guard); new bare-vs-full
case in `federation.kick.test.ts`; two new client-side cases in
`groupDm.ownerRouting.test.ts` covering both the extended-payload write path and
the legacy-server passthrough. Tests: 1053 server + 364 web, all green.
Specs updated: `dm-system.md` historical bugs + frontend handler table + WS
state-change events table; `federation.md` `ownership_transfer` receiver flow;
`websocket.md` event-fields table.
getProfileReferencedFilenames() didn't include dm_channels.icon, so the
storage janitor deleted group DM icons within ~1 hour:
- Owner instance: PATCH /api/dm/:id leaves an attachments row with
messageId=null and dmMessageId=null. After 1h, getUnlinkedAttachments
flags it and cleanupStorage phase 2 deletes the file because the
filename isn't in profileReferenced.
- Receiver instance: downloadProfileAsset writes the icon directly to
uploadDir with no attachments row. cleanupStorage phase 1 treats it
as orphaned and deletes it.
Fix: include dm_channels.icon (non-null, not soft-deleted, not http://)
in the profile-referenced set. Soft-deleted DMs are excluded so their
files still get reaped by cleanupSoftDeletedDmChannels. Absolute URLs
are skipped because they live on a remote instance.
Also mirror the avatar precedent at the PATCH endpoint by deleting the
new icon's standalone attachment row — the file is now protected via
dm_channels.icon, matching users.ts:473.
Two follow-on bugs from the initial S2S presence rollout:
(1) New friend stuck offline until they reload: presence_update fires only on
transitions, so a remote user already online when their stub is created
locally never receives a relay event seeding their actual status. The
stub defaulted to 'offline' at creation and stayed there until the next
transition. Fix: extend FederationRelayProfileSnapshot +
FederationUserLookupProfile with status. Sender-side buildProfileSnapshot,
getDmParticipants, and lookup endpoint responses populate it for native
users only (replicated stubs hold stale status owned elsewhere).
resolveOrCreateReplicatedUser uses hints.status to seed the new row's
status column. Threaded through every call site (DM participants, group
bootstrap, friend events, ownership transfer). Stub backfill worker also
heals existing rows whose status was stuck at 'offline' from creation.
(2) 'Online' text updates but green avatar dot stays grey on the same page:
spaceStore.updateMemberPresence patches members[] (which feeds space UIs)
but never patches userViews — the cache useCanonicalUserView reads from.
The Avatar in FriendItem reads canonical.status; the text reads
friend.status (socialStore). Two sources, one stale until full
user_updated arrives. Fix: updateMemberPresence now mirrors status into
matching userViews entries, so canonical-view consumers re-render with
fresh status the moment the WS event lands.
The outbox worker rebuilds FederationRelayEvent objects from stored JSON via
an allowlist of known fields. presenceUpdate was missed when presence_update
events shipped, so peers received events with eventType='presence_update' but
no payload — rejected with missing_presence_update_payload on every tick.
onPeerActivated re-emits a presence_update for every relationship-related
online native to the activating peer (relationship = friend with peer-stub /
DM-mate with peer-stub / replicatedInstances opt-in for peer origin). Scope
bounded by relationship count, not native count — flap recovery cost stays
proportional to actual interaction surface.
onPeerDeactivated flips every replicated stub from that peer to offline and
broadcasts a local presence_update so connected friends/DM-mates/space-co-members
see them disappear immediately, instead of seeing stale online until next signal.
Re-snapshot on every activation (incl. health-check unreachable→active flap)
is load-bearing for correctness — markPeerStubsOffline ran on the prior
deactivation and presence is not in the mutation log.
New FederationPresenceUpdatePayload + queuePresenceRelay() helper. Five WS
sites now project the native user's status (and optional activities) to all
active peers via the outbox: WS auth-success, finalizeDisconnect,
manual presence_update, activity_update, showActivity-toggle clear.
Outbox-only (no mutation-log entry) — presence is ephemeral; the upcoming
peer-activation hook re-emits a fresh snapshot so peers recovering from
unreachable converge without history replay. No-op for replicated users.
Heals existing legacy stubs (created when resolveOrCreateReplicatedUser used
homeUserId@domain) by asking the peer for the canonical username via
lookupRemoteUserByHomeId and rewriting the local row. Idempotent and
collision-safe.
Wired into onPeerActivated (per-origin) so future peer flaps re-attempt for
stubs whose home was unreachable on a prior pass, and into a one-shot pass at
startupBootstrapSync for all currently-active peers (not just first-time
lastSyncedAt=0 peers).
HMAC-authenticated, rate-limited (60/min/peer) endpoint that resolves a
homeUserId on this instance to its canonical username + profile snapshot.
Native non-deleted users only. Mirrors /users/lookup's auth shape.
Adds lookupRemoteUserByHomeId to federationLookup.ts as the client-side
helper. Used by the upcoming stub-backfill worker on peers that hold legacy
snowflake-named replicas of users now visible by their real handle.
Adds an explicit override for the federation transport URL returned by
getOurOrigin(). When unset, behaviour is unchanged (https://${DOMAIN} ->
http://localhost:${PORT} dev fallback). Intended for reverse-proxy /
dev-without-TLS deployments where the public origin must be advertised
explicitly (typically http://...) and differs from the bare DOMAIN
value used for federated identity.
Wired via config.publicOrigin (envOptional('PUBLIC_ORIGIN')) so the
override flows through the existing config layer rather than scattering
process.env reads. Trailing slash is stripped for symmetry with
peer.origin storage.
docs/systems/federation.md gets a "Public Origin Override" subsection
under §14 Background Workers documenting the resolution order.
hydrateReplicatedUserProfile now calls downloadProfileAsset and stores bare local filenames, falling back to absolute URLs only on download failure. It also fills empty fields only — no longer clobbering local files written by processProfileUpdateEvent. Adds an idempotent startup backfill that converts existing http-prefixed avatar/banner rows on replicated users into local files, so federated profile pictures keep rendering when the home instance is offline.
Adds an admin-driven sweep on top of the existing 24h auto-expire so
operators can see and reap abandoned `.tus/` sessions without waiting.
- storageJanitor: extract `walkTusDir(predicate)` helper, add
`getStaleTusInfo` + `cleanupStaleTusSessions(thresholdMs, dryRun)`;
refactor `cleanupTusStragglers` to delegate while preserving its
janitor-tick `{ removed }` contract.
- StorageStats gains `staleTusSessions` + `staleTusSize` (fixed 1h
display threshold).
- New `POST /api/admin/storage/cleanup-tus` route with
`maxAgeHours` validation (positive finite number, default 1) and
`dryRun` support; admin-gated.
- StoragePanel: 6th overview card "Stale Uploads" + new cleanup
subsection mirroring the media-cleanup pattern (preview-then-clean
with shared result panel styling).
- Tests: 8 new janitor tests covering empty dir, threshold filtering,
dry-run vs live, oldest-mtime tracking, subdir skipping, and the
override path on the existing straggler sweep. New
`routes/admin.test.ts` covers auth/admin gates, validation (zero,
negative, NaN), default `maxAgeHours`, dry-run vs live unlink.
- Docs: `uploads.md` §Janitor expanded to the full lifecycle (cancel
DELETE, discard DELETE, auto-expire, straggler sweep, admin route);
`admin.md` Storage Management updated with the new endpoint and
StorageStats fields.
Jannis Braun