Merge branch 'feat/peer-activation-recovery'

Closes #10b (S2S outbox sync recovery after peer state transitions).

Unifies three related bugs under a single on-peer-activation handler:
- Stranded outbox backoff after unreachable→active recovery
- Runtime sync only firing at startup (not on runtime peer re-creation)
- Silent enqueue failure for awaiting_approval / needs_attention peers

Plus:
- Mutation log coverage extended to dm_close/reopen, read_state_update,
  profile_update, and file_rejected (previously bypassed appendMutationLog)
- /api/federation/sync response builder gains serializers for those 5
  event types and a new contextType='profile' branch
- queueOutboxEvent rewritten with explicit per-status handling, mid-call
  race catch, and compile-time exhaustiveness check
- Pre-existing gap fixed: ensurePeered now handles needs_attention
  explicitly instead of falling through to auto-healing handshake

Verified end-to-end on Pi + VM across all three manual integration
scenarios (unreachable recovery, awaiting_approval drain, post-Reset
catch-up via mutation log).
This commit is contained in:
Jannis Braun
2026-04-22 01:35:32 +02:00
12 changed files with 1086 additions and 158 deletions
+73 -10
View File
@@ -977,16 +977,79 @@ The mutation log entry for reactions stores a simpler payload (no `messageId`/`m
## 12. Initial Sync ## 12. Initial Sync
### `runInitialSyncForNewPeers()` (`federationWorker.ts:739`) ### `startupBootstrapSync()` (`federationWorker.ts`)
Triggered once at server startup (async, non-blocking). Finds peers with `status = 'active'` and `lastSyncedAt = 0`. Triggered once at server startup (async, non-blocking). Finds peers with `status = 'active'` and `lastSyncedAt = 0` and calls `onPeerActivated(peerId, 'startup_bootstrap')` for each. This preserves the original startup-sync semantics while unifying the code path with all other activation sites (see "Peer Activation Recovery" below).
**For each unsynced peer:** ### Peer Activation Recovery
1. **DM sync pass:** Paginate through `POST {peerOrigin}/api/federation/sync` with `sinceTimestamp = 0`, `limit = 100`
2. **Direct processing:** Call `processRelayEvents()` to process received events in-process (no HTTP round-trip) Every transition of `federation_peers.status` to `active` invokes `onPeerActivated(peerId, reason)` — one handler wired at all transition sites. Two independent invariants, both unconditional:
3. **Friend sync pass:** Same pagination with `contextType: 'friend'`, also processed via `processRelayEvents()`
4. Update `lastSyncedAt = Date.now()` after completion 1. **`resetOutboxBackoff`** — sets `nextRetryAt = now` and `attempts = 0` for every outbox entry belonging to the peer. Entries that accumulated exponential backoff before the peer went unreachable are immediately eligible again. Attempts counter is also reset so a freshly-healthy peer's next failure starts at `BACKOFF_SCHEDULE_MS[0]` (30s), not wherever the counter left off.
5. On failure: don't update `lastSyncedAt` -- retried on next startup 2. **`syncPeerMutationLog`** — pulls missed events from the peer's `/api/federation/sync` endpoint since `peer.lastSyncedAt`. Three passes: DM, friend, profile (in that order, each paginated). `lastSyncedAt` advances to `Date.now()` on full success; stays put on transient failure so the next activation retries the same window.
#### Call sites (must remain exhaustive)
| File | Context | Reason |
|---|---|---|
| `routes/federation.ts` | `/peer/initiate` 200 activation | `initiate_accepted` |
| `routes/federation.ts` | `/peer/accept` existing-rejected override | `accept_rejected_override` |
| `routes/federation.ts` | `/peer/accept` existing-awaiting_approval | `accept_awaiting_approval` |
| `routes/federation.ts` | `/peer/accept` existing-pending | `accept_pending` |
| `routes/federation.ts` | `/peer/accept` new-peer | `accept_new` |
| `routes/federation.ts` | `/approval-requests/:id/approve` success | `approval_handshake` |
| `utils/federationWorker.ts` | Health check unreachable → active | `health_check_recovery` |
| `utils/federationPeering.ts` | `ensurePeered/performHandshake` 200 | `ensure_peered` |
| `utils/federationWorker.ts` | Startup scan (status=active, lastSyncedAt=0) | `startup_bootstrap` |
HTTP handler sites dispatch fire-and-forget (`.catch(log)`) so the response is not blocked by sync-pull pagination. Worker-internal sites `await` since the worker tick is already async.
Concurrent activations for the same peer are deduplicated via an in-flight promise map keyed by `peerId`.
#### Peer-state × outbox-enqueue × recovery matrix
| Status | `queueOutboxEvent` enqueue | Mutation log captures | Recovery on transition to `active` |
|---|---|---|---|
| `active` | Queue | Yes (for covered event types — see below) | N/A |
| `pending` | Queue | Yes | `onPeerActivated` |
| `unreachable` | Queue | Yes | `onPeerActivated` |
| `awaiting_approval` | **Drop, debug-log** | Yes | `onPeerActivated` |
| `needs_attention` | **Drop, debug-log** | Yes | `onPeerActivated` (fires when the row is re-created via admin Reset + re-peer) |
| `rejected` | Drop, debug-log | Yes | `onPeerActivated` |
| `revoked` | Drop, debug-log | Yes | `onPeerActivated` (fires when the row is re-created via hard-delete + re-initiate) |
`queueOutboxEvent` uses an exhaustive TypeScript `switch` on the narrowed peer-status union — adding a new status value without handling it fails compile-time typecheck (`const _exhaustive: never = status;`).
**Mid-call race catch:** when the initial peers SELECT filters out a peer because its status is non-deliverable, but the fallback loop observes the status has since flipped to `active`/`pending`/`unreachable`, the code re-fetches the peer row and appends it to `matchedPeers` so the outer enqueue loop includes it. Silent drops would lose real-time delivery under asymmetric failure (e.g., `/peer/accept` 200 response lost on the wire, health-check transition firing on only one side).
#### Mutation log coverage
Event types covered by `appendMutationLog` (replayed on sync-pull):
| Event type | `contextType` | Source |
|---|---|---|
| DM `create` / `update` / `delete` | `dm` | `federationOutbox.queueDmRelay`, `dm.ts` delete handler |
| `reaction_add` / `reaction_remove` | `dm` | `ws/events.ts` |
| `member_add` / `member_remove` / `ownership_transfer` | `dm` | `dm.ts` |
| `dm_close` / `dm_reopen` | `dm` | `federationOutbox.queueDmCloseRelay` |
| `read_state_update` | `dm` | `federationOutbox.queueReadStateRelay` |
| `file_rejected` | `dm` | `federationWorker.handleSizeRejection` |
| `friend_request_*` / `friend_add` / `friend_remove` | `friend` | `social.ts` |
| `profile_update` | `profile` | `routes/users.ts` (PATCH `/api/users/@me`) |
Ephemeral events (`dm_typing_*`, `dm_call_*`) are fire-and-forget by design and are NOT captured — missed typing/call-signaling packets are acceptable and carry no durable state.
#### `/api/federation/sync` contextType filter values
| Filter | Returns |
|---|---|
| (none) / omitted | DM events (including `dm_close`, `dm_reopen`, `read_state_update`, `file_rejected`) |
| `'friend'` | Friend events |
| `'profile'` | Profile update events |
#### Known Issues
- **Poison-pill event in peer's mutation log.** If a peer's mutation log contains a row whose inbound processor throws (e.g., a UNIQUE conflict from a malformed relay payload), `syncPeerMutationLog` catches the error and declines to advance `lastSyncedAt`. Subsequent activations retry the same window and hit the same failure, effectively blocking catch-up for that peer. No automatic poison-pill skip is implemented — recovery requires either: (a) fixing the mutation log on the peer side, or (b) manually advancing `lastSyncedAt` past the offending row via DB admin. Flagged as a follow-up backlog item.
### Sync Endpoint (`POST /api/federation/sync`) ### Sync Endpoint (`POST /api/federation/sync`)
@@ -994,7 +1057,7 @@ HMAC-authenticated. Returns events from the `federation_mutation_log`.
**Request:** **Request:**
```typescript ```typescript
{ sinceTimestamp: number, dmChannelId?: string, federatedId?: string, contextType?: 'dm'|'friend', limit?: 1-500 } { sinceTimestamp: number, dmChannelId?: string, federatedId?: string, contextType?: 'dm'|'friend'|'profile', limit?: 1-500 }
``` ```
**Response:** **Response:**
@@ -1125,7 +1188,7 @@ All workers are started by `startFederationWorkers()` on server boot and stopped
| File download | 30s | 5 | 60s | `processFileQueueTick` | | File download | 30s | 5 | 60s | `processFileQueueTick` |
| Health check | 15min | all unreachable | 10s | `processHealthCheckTick` | | Health check | 15min | all unreachable | 10s | `processHealthCheckTick` |
| Janitor | 1h | -- | -- | `runFederationJanitor` (sync) | | Janitor | 1h | -- | -- | `runFederationJanitor` (sync) |
| Initial sync | Once at startup | -- | 30s per page | `runInitialSyncForNewPeers` | | Startup bootstrap sync | Once at startup | -- | 30s per page | `startupBootstrapSync``onPeerActivated` |
### Janitor Cleanup (`storageJanitor.ts:runFederationJanitor`) ### Janitor Cleanup (`storageJanitor.ts:runFederationJanitor`)
+8 -6
View File
@@ -368,14 +368,16 @@ The sorted join ensures the same pair always produces the same prefix regardless
## 7. Initial Sync: Friend Backfill ## 7. Initial Sync: Friend Backfill
When a new peer is established (`federation_peers.lastSyncedAt = 0`), the federation worker runs `runInitialSyncForNewPeers()` on startup. This includes a dedicated friend sync pass. When a peer transitions to `active` (including at startup for peers with `lastSyncedAt = 0`), the federation worker calls `onPeerActivated(peerId, reason)`. One of its two unconditional invariants is `syncPeerMutationLog`, which pulls missed events from the peer's `/api/federation/sync` endpoint — including a dedicated friend sync pass.
**Flow (`federationWorker.ts:runInitialSyncForNewPeers`):** **Flow (`federationPeerActivation.ts:syncPeerMutationLog`):**
1. Query all active peers with `lastSyncedAt = 0` 1. **First pass (DM events):** Paginates through `POST /federation/sync` with no `contextType` filter (defaults to DM events), processing each batch via `processRelayEvents()` directly
2. **First pass (DM events):** Paginates through `POST /federation/sync` with no `contextType` filter (defaults to DM events), processing each batch via `processRelayEvents()` directly 2. **Second pass (friend events):** Paginates through `POST /federation/sync` with `contextType: 'friend'`, same direct processing
3. **Second pass (friend events):** Paginates through `POST /federation/sync` with `contextType: 'friend'`, same direct processing 3. **Third pass (profile events):** Paginates through `POST /federation/sync` with `contextType: 'profile'`, same direct processing
4. After both passes complete, updates `lastSyncedAt = Date.now()` so the sync doesn't repeat 4. After all three passes complete, updates `lastSyncedAt = Date.now()` so the window advances on the next activation
At startup, `startupBootstrapSync()` scans for `status = 'active' AND lastSyncedAt = 0` peers and calls `onPeerActivated(peerId, 'startup_bootstrap')` for each, preserving the original startup-sync semantics while using the unified path.
The sync endpoint (`POST /api/federation/sync`) returns events from the `federation_mutation_log` table, which retains entries for 90 days. This means friend relationships established within the last 90 days are backfilled when a new peer connection is created. The sync endpoint (`POST /api/federation/sync`) returns events from the `federation_mutation_log` table, which retains entries for 90 days. This means friend relationships established within the last 90 days are backfilled when a new peer connection is created.
+128 -5
View File
@@ -16,6 +16,7 @@ import { sanitizeUser } from '../utils/sanitize.js';
import { deleteAttachmentFiles, deleteUploadFile } from '../utils/fileCleanup.js'; import { deleteAttachmentFiles, deleteUploadFile } from '../utils/fileCleanup.js';
import { tombstoneUser, collectDeletionBroadcastTargets, collectProfileBroadcastTargetIds } from '../utils/userDeletion.js'; import { tombstoneUser, collectDeletionBroadcastTargets, collectProfileBroadcastTargetIds } from '../utils/userDeletion.js';
import { computeFederatedId, getDmParticipants, sendCallRelay } from '../utils/federationOutbox.js'; import { computeFederatedId, getDmParticipants, sendCallRelay } from '../utils/federationOutbox.js';
import { onPeerActivated } from '../utils/federationPeerActivation.js';
import { getDmMessageWithUser } from './dm.js'; import { getDmMessageWithUser } from './dm.js';
import type { FederationRelayRequest, FederationRelayResponse, FederationRelayEvent, FederationRelayAttachment, FederationSyncRequest, FederationSyncResponse, DmMessageWithUser, DmChannel, FederationRelayProfileSnapshot, FederationIdentityDeleteS2SRequest, FederationProfileUpdatePayload, ServerEvent } from '@backspace/shared'; import type { FederationRelayRequest, FederationRelayResponse, FederationRelayEvent, FederationRelayAttachment, FederationSyncRequest, FederationSyncResponse, DmMessageWithUser, DmChannel, FederationRelayProfileSnapshot, FederationIdentityDeleteS2SRequest, FederationProfileUpdatePayload, ServerEvent } from '@backspace/shared';
@@ -354,6 +355,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
.where(eq(schema.federationPeers.id, peerId)) .where(eq(schema.federationPeers.id, peerId))
.run(); .run();
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(peerId, 'initiate_accepted').catch(err =>
console.error('[federation] onPeerActivated from /peer/initiate failed:', err)
);
const peer = db const peer = db
.select() .select()
@@ -560,6 +564,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
} }
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(existing.id, 'accept_rejected_override').catch(err =>
console.error('[federation] onPeerActivated from /peer/accept (rejected override) failed:', err)
);
return reply.code(200).send({ accepted: true }); return reply.code(200).send({ accepted: true });
} }
@@ -583,6 +590,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
} }
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(existing.id, 'accept_awaiting_approval').catch(err =>
console.error('[federation] onPeerActivated from /peer/accept (awaiting_approval) failed:', err)
);
return reply.code(200).send({ accepted: true }); return reply.code(200).send({ accepted: true });
} }
@@ -597,6 +607,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
.run(); .run();
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(existing.id, 'accept_pending').catch(err =>
console.error('[federation] onPeerActivated from /peer/accept (pending) failed:', err)
);
return reply.code(200).send({ accepted: true }); return reply.code(200).send({ accepted: true });
} }
@@ -613,6 +626,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
}).run(); }).run();
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(peerId, 'accept_new').catch(err =>
console.error('[federation] onPeerActivated from /peer/accept (new) failed:', err)
);
return reply.code(200).send({ accepted: true }); return reply.code(200).send({ accepted: true });
}, },
@@ -1136,6 +1152,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
.run(); .run();
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(peerId, 'approval_handshake').catch(err =>
console.error('[federation] onPeerActivated from /approval-requests/:id/approve failed:', err)
);
const peer = db const peer = db
.select() .select()
@@ -1615,6 +1634,10 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
payload: string | null; payload: string | null;
}>; }>;
// Maps local DM channel ID → federatedId for O(1) lookup in serializers.
// Only populated in the DM branch (friend/profile branches don't need it).
let channelFederatedIdMap = new Map<string, string>();
if (contextTypeFilter === 'friend') { if (contextTypeFilter === 'friend') {
// ── Friend event sync: no DM channel logic needed ── // ── Friend event sync: no DM channel logic needed ──
mutationRows = rawDb.prepare(` mutationRows = rawDb.prepare(`
@@ -1624,6 +1647,15 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
ORDER BY mutated_at ASC ORDER BY mutated_at ASC
LIMIT ? LIMIT ?
`).all(sinceTimestamp, limit) as typeof mutationRows; `).all(sinceTimestamp, limit) as typeof mutationRows;
} else if (contextTypeFilter === 'profile') {
// ── Profile event sync: no DM channel logic needed ──
mutationRows = rawDb.prepare(`
SELECT id, entity_id, context_id, context_type, mutation_type, mutated_at, payload
FROM federation_mutation_log
WHERE context_type = 'profile' AND mutated_at > ?
ORDER BY mutated_at ASC
LIMIT ?
`).all(sinceTimestamp, limit) as typeof mutationRows;
} else { } else {
// ── DM sync path ── // ── DM sync path ──
// Determine which DM channels to sync. // Determine which DM channels to sync.
@@ -1631,11 +1663,14 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
// that should be synced. The peer's relay endpoint will create the channel // that should be synced. The peer's relay endpoint will create the channel
// if it doesn't exist, or match by federated_id if it does. // if it doesn't exist, or match by federated_id if it does.
const sharedChannelRows = rawDb.prepare(` const sharedChannelRows = rawDb.prepare(`
SELECT id as dm_channel_id FROM dm_channels SELECT id as dm_channel_id, federated_id FROM dm_channels
WHERE federated_id IS NOT NULL AND deleted_at IS NULL WHERE federated_id IS NOT NULL AND deleted_at IS NULL
`).all() as Array<{ dm_channel_id: string }>; `).all() as Array<{ dm_channel_id: string; federated_id: string }>;
const sharedChannelIds = sharedChannelRows.map(r => r.dm_channel_id); const sharedChannelIds = sharedChannelRows.map(r => r.dm_channel_id);
channelFederatedIdMap = new Map<string, string>(
sharedChannelRows.map(r => [r.dm_channel_id, r.federated_id])
);
// If filtering by federatedId, resolve to local channel ID // If filtering by federatedId, resolve to local channel ID
let effectiveChannelFilter = dmChannelIdFilter; let effectiveChannelFilter = dmChannelIdFilter;
@@ -1679,7 +1714,10 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
WHERE ml.context_id = ? WHERE ml.context_id = ?
AND ml.context_type = 'dm' AND ml.context_type = 'dm'
AND ml.mutated_at > ? AND ml.mutated_at > ?
AND (dm.id IS NOT NULL OR ml.mutation_type IN ('delete', 'member_add', 'member_remove', 'ownership_transfer')) AND (dm.id IS NOT NULL OR ml.mutation_type IN (
'delete', 'member_add', 'member_remove', 'ownership_transfer',
'dm_close', 'dm_reopen', 'read_state_update', 'file_rejected'
))
ORDER BY ml.mutated_at ASC ORDER BY ml.mutated_at ASC
LIMIT ? LIMIT ?
`).all(effectiveChannelFilter, sinceTimestamp, limit) as typeof mutationRows; `).all(effectiveChannelFilter, sinceTimestamp, limit) as typeof mutationRows;
@@ -1720,7 +1758,10 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
WHERE ml.context_id IN (${placeholders}) WHERE ml.context_id IN (${placeholders})
AND ml.context_type = 'dm' AND ml.context_type = 'dm'
AND ml.mutated_at > ? AND ml.mutated_at > ?
AND (dm.id IS NOT NULL OR ml.mutation_type IN ('delete', 'member_add', 'member_remove', 'ownership_transfer')) AND (dm.id IS NOT NULL OR ml.mutation_type IN (
'delete', 'member_add', 'member_remove', 'ownership_transfer',
'dm_close', 'dm_reopen', 'read_state_update', 'file_rejected'
))
ORDER BY ml.mutated_at ASC ORDER BY ml.mutated_at ASC
LIMIT ? LIMIT ?
`).all(...sharedChannelIds, sinceTimestamp, limit) as typeof mutationRows; `).all(...sharedChannelIds, sinceTimestamp, limit) as typeof mutationRows;
@@ -1760,7 +1801,9 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
const mutationType = mutation.mutation_type as 'create' | 'update' | 'delete' | 'reaction_add' | 'reaction_remove' const mutationType = mutation.mutation_type as 'create' | 'update' | 'delete' | 'reaction_add' | 'reaction_remove'
| 'member_add' | 'member_remove' | 'ownership_transfer' | 'member_add' | 'member_remove' | 'ownership_transfer'
| 'friend_request_create' | 'friend_request_update' | 'friend_request_cancel' | 'friend_request_create' | 'friend_request_update' | 'friend_request_cancel'
| 'friend_add' | 'friend_remove'; | 'friend_add' | 'friend_remove'
| 'dm_close' | 'dm_reopen' | 'read_state_update' | 'file_rejected'
| 'profile_update';
if (['member_add', 'member_remove', 'ownership_transfer', if (['member_add', 'member_remove', 'ownership_transfer',
'friend_request_create', 'friend_request_update', 'friend_request_cancel', 'friend_request_create', 'friend_request_update', 'friend_request_cancel',
@@ -1820,6 +1863,86 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
continue; continue;
} }
if (mutationType === 'dm_close' || mutationType === 'dm_reopen') {
if (!mutation.payload) continue;
let dmCloseReopenPayload: { homeUserId: string; homeInstance: string } | null = null;
try { dmCloseReopenPayload = JSON.parse(mutation.payload); } catch { continue; }
if (!dmCloseReopenPayload) continue;
const fedIdCloseReopen = channelFederatedIdMap.get(mutation.context_id);
if (!fedIdCloseReopen) continue;
events.push({
eventType: mutationType,
dmChannelId: mutation.context_id,
messageId: mutation.entity_id,
federatedId: fedIdCloseReopen,
encryptionVersion: 0,
timestamp: mutation.mutated_at,
dmCloseReopen: dmCloseReopenPayload,
});
continue;
}
if (mutationType === 'read_state_update') {
if (!mutation.payload) continue;
let readState: NonNullable<FederationRelayEvent['readState']> | null = null;
try { readState = JSON.parse(mutation.payload) as NonNullable<FederationRelayEvent['readState']>; } catch { continue; }
if (!readState) continue;
const fedIdReadState = channelFederatedIdMap.get(mutation.context_id);
if (!fedIdReadState) continue;
events.push({
eventType: 'read_state_update',
dmChannelId: mutation.context_id,
messageId: mutation.entity_id,
federatedId: fedIdReadState,
encryptionVersion: 0,
timestamp: mutation.mutated_at,
readState,
});
continue;
}
if (mutationType === 'file_rejected') {
if (!mutation.payload) continue;
let fileRejectedPayload: {
attachmentId: string;
sourceFilename: string;
rejectionReason: string;
rejectionLimit: number;
affectedUserIds: string[];
} | null = null;
try { fileRejectedPayload = JSON.parse(mutation.payload); } catch { continue; }
if (!fileRejectedPayload) continue;
events.push({
eventType: 'file_rejected',
dmChannelId: mutation.context_id,
messageId: mutation.entity_id,
encryptionVersion: 0,
timestamp: mutation.mutated_at,
attachmentId: fileRejectedPayload.attachmentId,
sourceFilename: fileRejectedPayload.sourceFilename,
rejectionReason: fileRejectedPayload.rejectionReason,
rejectionLimit: fileRejectedPayload.rejectionLimit,
affectedUserIds: fileRejectedPayload.affectedUserIds,
});
continue;
}
if (mutationType === 'profile_update') {
if (!mutation.payload) continue;
let profileOuter: { profileUpdate?: NonNullable<FederationRelayEvent['profileUpdate']> } | null = null;
try { profileOuter = JSON.parse(mutation.payload); } catch { continue; }
if (!profileOuter?.profileUpdate) continue;
events.push({
eventType: 'profile_update',
contextType: 'profile',
messageId: mutation.entity_id,
encryptionVersion: 0,
timestamp: mutation.mutated_at,
profileUpdate: profileOuter.profileUpdate,
});
continue;
}
// For create and update: fetch the current message state // For create and update: fetch the current message state
const message = db const message = db
.select() .select()
+15 -1
View File
@@ -8,7 +8,7 @@ import { AVATAR_COLORS } from '@backspace/shared';
import { sanitizeUser } from '../utils/sanitize.js'; import { sanitizeUser } from '../utils/sanitize.js';
import { deleteUploadFile, deleteAttachmentByFilename } from '../utils/fileCleanup.js'; import { deleteUploadFile, deleteAttachmentByFilename } from '../utils/fileCleanup.js';
import { tombstoneUser, collectDeletionBroadcastTargets, collectProfileBroadcastTargetIds } from '../utils/userDeletion.js'; import { tombstoneUser, collectDeletionBroadcastTargets, collectProfileBroadcastTargetIds } from '../utils/userDeletion.js';
import { queueOutboxEvent, isFederationRelayEnabled } from '../utils/federationOutbox.js'; import { queueOutboxEvent, isFederationRelayEnabled, appendMutationLog } from '../utils/federationOutbox.js';
import { generateSnowflake } from '../utils/snowflake.js'; import { generateSnowflake } from '../utils/snowflake.js';
import { resizeProfileImage } from '../utils/thumbnail.js'; import { resizeProfileImage } from '../utils/thumbnail.js';
import { config } from '../config.js'; import { config } from '../config.js';
@@ -374,6 +374,13 @@ export async function userRoutes(app: FastifyInstance): Promise<void> {
bio: preUpdateUser.bio, bio: preUpdateUser.bio,
}; };
appendMutationLog(
preUpdateUser.id,
preUpdateUser.id,
'profile_update',
JSON.stringify({ profileUpdate: profilePayload }),
'profile',
);
for (const targetOrigin of newOrigins) { for (const targetOrigin of newOrigins) {
queueOutboxEvent( queueOutboxEvent(
preUpdateUser.id, preUpdateUser.id,
@@ -529,6 +536,13 @@ export async function userRoutes(app: FastifyInstance): Promise<void> {
bio: updatedUser!.bio, bio: updatedUser!.bio,
}; };
appendMutationLog(
updatedUser!.id,
updatedUser!.id,
'profile_update',
JSON.stringify({ profileUpdate: profilePayload }),
'profile',
);
queueOutboxEvent( queueOutboxEvent(
updatedUser!.id, // entityId — user's ID (coalesces rapid edits) updatedUser!.id, // entityId — user's ID (coalesces rapid edits)
updatedUser!.id, // contextId — user-scoped updatedUser!.id, // contextId — user-scoped
@@ -0,0 +1,135 @@
import { describe, it, expect, beforeEach, vi } from 'vitest';
import Database from 'better-sqlite3';
import { drizzle } from 'drizzle-orm/better-sqlite3';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import * as schema from '../db/schema.js';
import { eq } from 'drizzle-orm';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
type TestDb = ReturnType<typeof drizzle<typeof schema>>;
let sqlite: Database.Database;
let testDb: TestDb;
vi.mock('../db/index.js', () => ({
getDb: () => testDb,
schema,
}));
vi.mock('../utils/federationAuth.js', () => ({
getOurOrigin: () => 'https://test.example',
buildFederationHeaders: () => ({}),
generateHmacSecret: () => 'secret',
}));
let _snowflakeCounter = 1;
vi.mock('../utils/snowflake.js', () => ({
generateSnowflake: () => String(_snowflakeCounter++),
setWorkerId: vi.fn(),
}));
function applyMigrations(db: Database.Database): void {
const migrationsDir = path.resolve(__dirname, '../../drizzle');
const files = fs.readdirSync(migrationsDir).filter(f => f.endsWith('.sql')).sort();
for (const f of files) {
const sql = fs.readFileSync(path.join(migrationsDir, f), 'utf8');
const statements = sql.split(/-->\s*statement-breakpoint/);
for (const stmt of statements) {
const clean = stmt.trim();
if (clean) db.exec(clean);
}
}
}
function seedSettings(): void {
testDb.insert(schema.instanceSettings).values({
id: 1,
federationRelayEnabled: 1,
updatedAt: Date.now(),
}).run();
}
function seedChannel(id: string, federatedId: string | null): void {
testDb.insert(schema.dmChannels).values({
id, federatedId, ownerId: null, createdAt: Date.now(),
}).run();
}
function seedUser(id: string, username: string): void {
testDb.insert(schema.users).values({
id, username, displayName: username, passwordHash: 'x',
createdAt: Date.now(),
}).run();
}
function seedDmMember(channelId: string, userId: string): void {
testDb.insert(schema.dmMembers).values({
dmChannelId: channelId, userId, closed: 0,
}).run();
}
describe('queueDmCloseRelay — mutation log capture', () => {
beforeEach(() => {
sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
seedSettings();
});
it('appends a mutation log row for dm_close', async () => {
const { queueDmCloseRelay } = await import('./federationOutbox.js');
seedUser('u-1', 'alice');
seedChannel('ch-1', 'fed-1');
seedDmMember('ch-1', 'u-1');
queueDmCloseRelay('ch-1', 'u-1', 'dm_close');
const rows = testDb.select().from(schema.federationMutationLog)
.where(eq(schema.federationMutationLog.mutationType, 'dm_close')).all();
expect(rows.length).toBe(1);
expect(rows[0]?.contextId).toBe('ch-1');
expect(rows[0]?.contextType).toBe('dm');
});
it('appends a mutation log row for dm_reopen', async () => {
const { queueDmCloseRelay } = await import('./federationOutbox.js');
seedUser('u-1', 'alice');
seedChannel('ch-2', 'fed-2');
seedDmMember('ch-2', 'u-1');
queueDmCloseRelay('ch-2', 'u-1', 'dm_reopen');
const rows = testDb.select().from(schema.federationMutationLog)
.where(eq(schema.federationMutationLog.mutationType, 'dm_reopen')).all();
expect(rows.length).toBe(1);
});
});
describe('queueReadStateRelay — mutation log capture', () => {
beforeEach(() => {
sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
seedSettings();
});
it('appends a mutation log row for read_state_update', async () => {
const { queueReadStateRelay } = await import('./federationOutbox.js');
seedUser('u-2', 'bob');
seedChannel('ch-3', 'fed-3');
seedDmMember('ch-3', 'u-2');
testDb.insert(schema.dmMessages).values({
id: 'm-1', dmChannelId: 'ch-3', userId: 'u-2', content: 'hi',
type: 'user', createdAt: Date.now(),
}).run();
queueReadStateRelay('ch-3', 'm-1', 'u-2');
const rows = testDb.select().from(schema.federationMutationLog)
.where(eq(schema.federationMutationLog.mutationType, 'read_state_update')).all();
expect(rows.length).toBe(1);
expect(rows[0]?.contextId).toBe('ch-3');
expect(rows[0]?.contextType).toBe('dm');
});
});
@@ -0,0 +1,92 @@
import { describe, it, expect, beforeEach, vi } from 'vitest';
import Database from 'better-sqlite3';
import { drizzle } from 'drizzle-orm/better-sqlite3';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import * as schema from '../db/schema.js';
import { eq } from 'drizzle-orm';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
type TestDb = ReturnType<typeof drizzle<typeof schema>>;
// Mutable reference updated in beforeEach — the factory closes over this.
let testDb: TestDb;
vi.mock('../db/index.js', () => ({
getDb: () => testDb,
schema,
}));
// Mock federation-auth helpers to avoid env-var dependency
vi.mock('../utils/federationAuth.js', () => ({
getOurOrigin: () => 'https://local.example',
buildFederationHeaders: () => ({}),
generateHmacSecret: () => 'test-secret',
}));
function applyMigrations(db: Database.Database): void {
const migrationsDir = path.resolve(__dirname, '../../drizzle');
const files = fs.readdirSync(migrationsDir).filter(f => f.endsWith('.sql')).sort();
for (const f of files) {
const sql = fs.readFileSync(path.join(migrationsDir, f), 'utf8');
const statements = sql.split(/-->\s*statement-breakpoint/);
for (const stmt of statements) {
const clean = stmt.trim();
if (clean) db.exec(clean);
}
}
}
function seedSettings(): void {
testDb.insert(schema.instanceSettings).values({
id: 1,
federationRelayEnabled: 1,
federationRelayTtlDays: 30,
updatedAt: Date.now(),
}).run();
}
function seedPeer(id: string, origin: string, status: string): void {
testDb.insert(schema.federationPeers).values({
id, origin, hmacSecret: 'secret',
status, lastSyncedAt: 0, createdAt: Date.now(),
}).run();
}
function countOutbox(peerId: string): number {
return testDb.select().from(schema.federationOutbox)
.where(eq(schema.federationOutbox.peerId, peerId))
.all().length;
}
// Import once at module level — vi.mock is hoisted and the factory returns the
// live testDb reference, so re-using the cached import is correct.
const { queueOutboxEvent } = await import('./federationOutbox.js');
describe('queueOutboxEvent — non-deliverable statuses', () => {
beforeEach(() => {
const sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
seedSettings();
vi.restoreAllMocks();
});
it.each([
['awaiting_approval'],
['needs_attention'],
['rejected'],
['revoked'],
])('drops the event and logs a reason for %s peers (no outbox row, no throw)', (status) => {
seedPeer('peer-drop', 'https://drop.example', status);
const debugSpy = vi.spyOn(console, 'debug').mockImplementation(() => {});
queueOutboxEvent('entity-1', 'ctx-1', 'create', '{}', ['https://drop.example'], 'dm');
expect(countOutbox('peer-drop')).toBe(0);
expect(debugSpy).toHaveBeenCalled();
expect(debugSpy.mock.calls[0]![0] as string).toContain(status);
});
});
+77 -21
View File
@@ -163,40 +163,84 @@ export function queueOutboxEvent(
for (const origin of targetPeerOrigins) { for (const origin of targetPeerOrigins) {
if (matchedOrigins.has(origin)) continue; if (matchedOrigins.has(origin)) continue;
// Check if there's a rejected/revoked peer we should skip
const existingPeer = db const existingPeer = db
.select({ status: schema.federationPeers.status }) .select({ status: schema.federationPeers.status })
.from(schema.federationPeers) .from(schema.federationPeers)
.where(eq(schema.federationPeers.origin, origin)) .where(eq(schema.federationPeers.origin, origin))
.get(); .get();
if (existingPeer && (existingPeer.status === 'rejected' || existingPeer.status === 'revoked')) { if (!existingPeer) {
console.warn(`[federation] queueOutboxEvent: skipping ${existingPeer.status} peer ${origin}`); // No peer row — create pending placeholder, handshake fires on next tick
continue; const peerId = generateSnowflake();
} const now = Date.now();
db.insert(schema.federationPeers).values({
// No peer record at all — create a pending placeholder
const peerId = generateSnowflake();
const now = Date.now();
db.insert(schema.federationPeers)
.values({
id: peerId, id: peerId,
origin, origin,
hmacSecret: generateHmacSecret(), hmacSecret: generateHmacSecret(),
status: 'pending', status: 'pending',
createdAt: now, createdAt: now,
}) }).run();
.run(); const newPeer = db.select().from(schema.federationPeers)
.where(eq(schema.federationPeers.id, peerId)).get();
if (newPeer) {
matchedPeers = [...matchedPeers, newPeer];
console.log(`[federation] queueOutboxEvent: created pending placeholder for ${origin}`);
}
continue;
}
const newPeer = db // schema.federationPeers.status is plain text — narrow to known union for
.select() // compile-time exhaustiveness check without widening to `string`.
.from(schema.federationPeers) const status = existingPeer.status as
.where(eq(schema.federationPeers.id, peerId)) | 'active'
.get(); | 'pending'
| 'unreachable'
| 'awaiting_approval'
| 'needs_attention'
| 'rejected'
| 'revoked';
if (newPeer) { switch (status) {
matchedPeers = [...matchedPeers, newPeer]; case 'active':
console.log(`[federation] queueOutboxEvent: created pending placeholder for ${origin}`); case 'pending':
case 'unreachable': {
// Race: peer transitioned to a deliverable status between the initial
// peers SELECT and this point in the loop. Re-fetch the full row and
// add to matchedPeers so the outer enqueue loop includes this peer.
// Do NOT silently drop — symmetric onPeerActivated on the peer's side
// is not guaranteed to cover asymmetric-failure cases (lost /peer/accept
// 200, health-check-only transition on one side).
const raced = db
.select()
.from(schema.federationPeers)
.where(eq(schema.federationPeers.origin, origin))
.get();
if (raced) {
matchedPeers = [...matchedPeers, raced];
console.log(`[federation] queueOutboxEvent: race-caught ${origin} (now ${status}); enqueueing`);
}
break;
}
case 'awaiting_approval':
console.debug(`[federation] queueOutboxEvent: skipping ${origin} (awaiting_approval); mutation log will replay on activation`);
break;
case 'needs_attention':
console.debug(`[federation] queueOutboxEvent: skipping ${origin} (needs_attention; admin Reset required); mutation log will replay after Reset + re-peer`);
break;
case 'rejected':
console.debug(`[federation] queueOutboxEvent: skipping ${origin} (rejected peering)`);
break;
case 'revoked':
console.debug(`[federation] queueOutboxEvent: skipping ${origin} (revoked by admin)`);
break;
default: {
// Exhaustiveness check — no `as never` cast. TypeScript enforces
// that every status value is handled; adding a new value to the
// union without a case here fails typecheck.
const _exhaustive: never = status;
console.error(`[federation] queueOutboxEvent: unknown peer status for ${origin}: ${String(_exhaustive)}`);
break;
}
} }
} }
} }
@@ -656,6 +700,12 @@ export function queueReadStateRelay(
}; };
const targetOrigins = getGroupDmTargetOrigins(channelId); const targetOrigins = getGroupDmTargetOrigins(channelId);
appendMutationLog(
`read_state:${channel.federatedId}:${userId}`,
channelId,
'read_state_update',
JSON.stringify({ user: { homeUserId, homeInstance }, messageRef }),
);
queueOutboxEvent( queueOutboxEvent(
`read_state:${channel.federatedId}:${userId}`, `read_state:${channel.federatedId}:${userId}`,
channelId, channelId,
@@ -711,6 +761,12 @@ export function queueDmCloseRelay(
}; };
const targetOrigins = getGroupDmTargetOrigins(dmChannelId); const targetOrigins = getGroupDmTargetOrigins(dmChannelId);
appendMutationLog(
`${eventType}:${channel.federatedId}:${userId}`,
dmChannelId,
eventType,
JSON.stringify({ homeUserId, homeInstance }),
);
queueOutboxEvent( queueOutboxEvent(
`${eventType}:${channel.federatedId}:${userId}`, `${eventType}:${channel.federatedId}:${userId}`,
dmChannelId, dmChannelId,
@@ -0,0 +1,296 @@
import { describe, it, expect, beforeEach, vi } from 'vitest';
import Database from 'better-sqlite3';
import { drizzle } from 'drizzle-orm/better-sqlite3';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import * as schema from '../db/schema.js';
import { eq } from 'drizzle-orm';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
type TestDb = ReturnType<typeof drizzle<typeof schema>>;
let sqlite: Database.Database;
let testDb: TestDb;
vi.mock('../db/index.js', () => ({
getDb: () => testDb,
schema,
}));
vi.mock('../utils/federationOutbox.js', () => ({
isFederationRelayEnabled: () => true,
}));
vi.mock('../utils/federationAuth.js', () => ({
getOurOrigin: () => 'https://local.example',
buildFederationHeaders: (_body: string, _secret: string, _origin: string) => ({
'Content-Type': 'application/json',
'X-Federation-Origin': _origin,
}),
}));
vi.mock('../routes/federation.js', () => ({
processRelayEvents: vi.fn().mockResolvedValue({ accepted: [], rejected: [] }),
}));
vi.mock('../ws/handler.js', () => ({
connectionManager: {
sendToAdmins: vi.fn(),
getAllOnlineUserIds: () => [],
sendToUser: vi.fn(),
sendToDmMembers: vi.fn(),
},
}));
function applyMigrations(db: Database.Database): void {
const migrationsDir = path.resolve(__dirname, '../../drizzle');
const files = fs.readdirSync(migrationsDir).filter(f => f.endsWith('.sql')).sort();
for (const f of files) {
const sql = fs.readFileSync(path.join(migrationsDir, f), 'utf8');
const statements = sql.split(/-->\s*statement-breakpoint/);
for (const stmt of statements) {
const clean = stmt.trim();
if (clean) db.exec(clean);
}
}
}
function seedPeer(id: string, status: string, lastSyncedAt = 0): void {
testDb.insert(schema.federationPeers).values({
id, origin: `https://${id}.example`, hmacSecret: 'secret',
status, lastSyncedAt, createdAt: Date.now(),
}).run();
}
function seedOutboxEntry(id: string, peerId: string, nextRetryAt: number, attempts: number): void {
testDb.insert(schema.federationOutbox).values({
id, peerId, contextId: 'ch-1', entityId: `msg-${id}`,
contextType: 'dm', eventType: 'create', payload: '{}',
encryptionVersion: 0, attempts, nextRetryAt,
expiresAt: Date.now() + 30 * 86_400_000,
createdAt: Date.now(),
}).run();
}
describe('resetOutboxBackoff', () => {
beforeEach(() => {
sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
});
it('resets nextRetryAt=now and attempts=0 for all peer entries — including past-due ones', async () => {
const { resetOutboxBackoff } = await import('./federationPeerActivation.js');
seedPeer('peer-a', 'active');
seedPeer('peer-b', 'active');
const now = Date.now();
// Three entries for peer-a: past-due (already eligible), near-future, far-future
seedOutboxEntry('entry-1', 'peer-a', now - 1000, 5);
seedOutboxEntry('entry-2', 'peer-a', now + 60_000, 3);
seedOutboxEntry('entry-3', 'peer-a', now + 86_400_000, 7);
// Entry for unrelated peer-b (must NOT be touched)
seedOutboxEntry('entry-4', 'peer-b', now + 86_400_000, 9);
resetOutboxBackoff('peer-a');
const a1 = testDb.select().from(schema.federationOutbox).where(eq(schema.federationOutbox.id, 'entry-1')).get();
const a2 = testDb.select().from(schema.federationOutbox).where(eq(schema.federationOutbox.id, 'entry-2')).get();
const a3 = testDb.select().from(schema.federationOutbox).where(eq(schema.federationOutbox.id, 'entry-3')).get();
const b4 = testDb.select().from(schema.federationOutbox).where(eq(schema.federationOutbox.id, 'entry-4')).get();
// All peer-a entries reset — including the past-due one (correctness: attempts=0 on those too)
expect(a1?.attempts).toBe(0);
expect(a2?.attempts).toBe(0);
expect(a3?.attempts).toBe(0);
expect(a1?.nextRetryAt).toBeGreaterThanOrEqual(now);
expect(a2?.nextRetryAt).toBeLessThanOrEqual(Date.now());
expect(a3?.nextRetryAt).toBeLessThanOrEqual(Date.now());
// peer-b untouched
expect(b4?.attempts).toBe(9);
expect(b4?.nextRetryAt).toBe(now + 86_400_000);
});
it('is a no-op when the peer has no outbox entries', async () => {
const { resetOutboxBackoff } = await import('./federationPeerActivation.js');
seedPeer('peer-empty', 'active');
expect(() => resetOutboxBackoff('peer-empty')).not.toThrow();
});
});
describe('syncPeerMutationLog', () => {
beforeEach(() => {
sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
vi.restoreAllMocks();
});
it('seeds sinceTimestamp from peer.lastSyncedAt for each pass', async () => {
const { syncPeerMutationLog } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-1', origin: 'https://peer-1.example', hmacSecret: 'secret',
status: 'active', lastSyncedAt: 5000, createdAt: Date.now(),
}).run();
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(async () =>
new Response(JSON.stringify({ events: [], hasMore: false, checkpoint: 5000 }), { status: 200 })
);
await syncPeerMutationLog('peer-1', 'health_check_recovery');
// Three passes: dm (no contextType), friend, profile
expect(fetchSpy).toHaveBeenCalledTimes(3);
for (const call of fetchSpy.mock.calls) {
const body = JSON.parse(call[1]?.body as string) as { sinceTimestamp: number };
expect(body.sinceTimestamp).toBe(5000);
}
const calls = fetchSpy.mock.calls.map(c => JSON.parse(c[1]?.body as string) as { contextType?: string });
expect(calls[0]!.contextType).toBeUndefined(); // DM pass (no contextType filter)
expect(calls[1]!.contextType).toBe('friend');
expect(calls[2]!.contextType).toBe('profile');
});
it('advances lastSyncedAt on success', async () => {
const { syncPeerMutationLog } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-2', origin: 'https://peer-2.example', hmacSecret: 'secret',
status: 'active', lastSyncedAt: 0, createdAt: Date.now(),
}).run();
vi.spyOn(globalThis, 'fetch').mockImplementation(async () =>
new Response(JSON.stringify({ events: [], hasMore: false, checkpoint: 1000 }), { status: 200 })
);
const before = Date.now();
await syncPeerMutationLog('peer-2', 'startup_bootstrap');
const after = Date.now();
const row = testDb.select().from(schema.federationPeers)
.where(eq(schema.federationPeers.id, 'peer-2')).get();
expect(row?.lastSyncedAt).toBeGreaterThanOrEqual(before);
expect(row?.lastSyncedAt).toBeLessThanOrEqual(after);
});
it('does NOT update lastSyncedAt on transient failure', async () => {
const { syncPeerMutationLog } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-3', origin: 'https://peer-3.example', hmacSecret: 'secret',
status: 'active', lastSyncedAt: 42_000, createdAt: Date.now(),
}).run();
vi.spyOn(globalThis, 'fetch').mockImplementation(async () =>
new Response('internal error', { status: 500 })
);
await syncPeerMutationLog('peer-3', 'ensure_peered');
const row = testDb.select().from(schema.federationPeers)
.where(eq(schema.federationPeers.id, 'peer-3')).get();
expect(row?.lastSyncedAt).toBe(42_000);
});
it('does nothing when peer is not active', async () => {
const { syncPeerMutationLog } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-4', origin: 'https://peer-4.example', hmacSecret: 'secret',
status: 'pending', lastSyncedAt: 0, createdAt: Date.now(),
}).run();
const fetchSpy = vi.spyOn(globalThis, 'fetch');
await syncPeerMutationLog('peer-4', 'health_check_recovery');
expect(fetchSpy).not.toHaveBeenCalled();
});
it('advances sinceTimestamp within a pass using data.checkpoint when hasMore is true', async () => {
const { syncPeerMutationLog } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-5', origin: 'https://peer-5.example', hmacSecret: 'secret',
status: 'active', lastSyncedAt: 100, createdAt: Date.now(),
}).run();
let call = 0;
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(async () => {
call++;
// First DM call: one event, hasMore=true, checkpoint advances to 2500
// Second DM call: empty, hasMore=false, ends the DM pass
// Remaining calls (friend, profile): empty/done immediately
if (call === 1) {
return new Response(
JSON.stringify({
events: [{ eventType: 'create', messageId: 'm1', timestamp: 200, encryptionVersion: 0 }],
hasMore: true,
checkpoint: 2500,
}),
{ status: 200 },
);
}
return new Response(JSON.stringify({ events: [], hasMore: false, checkpoint: call === 2 ? 2500 : 100 }), { status: 200 });
});
await syncPeerMutationLog('peer-5', 'health_check_recovery');
// Call 1: DM pass, since=100 (peer.lastSyncedAt)
// Call 2: DM pass continuation, since=2500 (advanced by previous checkpoint)
// Call 3: friend pass, since=100 (re-seeded from peer.lastSyncedAt)
// Call 4: profile pass, since=100 (re-seeded from peer.lastSyncedAt)
expect(fetchSpy).toHaveBeenCalledTimes(4);
const bodies = fetchSpy.mock.calls.map(c => JSON.parse(c[1]?.body as string) as { sinceTimestamp: number; contextType?: string });
expect(bodies[0]?.sinceTimestamp).toBe(100);
expect(bodies[0]?.contextType).toBeUndefined();
expect(bodies[1]?.sinceTimestamp).toBe(2500); // advanced by checkpoint from call 1
expect(bodies[1]?.contextType).toBeUndefined();
expect(bodies[2]?.sinceTimestamp).toBe(100); // friend pass re-seeds from peer.lastSyncedAt
expect(bodies[2]?.contextType).toBe('friend');
expect(bodies[3]?.sinceTimestamp).toBe(100); // profile pass re-seeds from peer.lastSyncedAt
expect(bodies[3]?.contextType).toBe('profile');
});
});
describe('onPeerActivated', () => {
beforeEach(() => {
sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
vi.restoreAllMocks();
});
it('runs resetOutboxBackoff and syncPeerMutationLog once, even under concurrent calls', async () => {
const { onPeerActivated } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-x', origin: 'https://peer-x.example', hmacSecret: 'secret',
status: 'active', lastSyncedAt: 0, createdAt: Date.now(),
}).run();
let fetchCount = 0;
vi.spyOn(globalThis, 'fetch').mockImplementation(async () => {
fetchCount++;
// Deliberately slow to let the second concurrent call share the in-flight promise.
await new Promise(r => setTimeout(r, 20));
return new Response(JSON.stringify({ events: [], hasMore: false, checkpoint: 0 }), { status: 200 });
});
const p1 = onPeerActivated('peer-x', 'health_check_recovery');
const p2 = onPeerActivated('peer-x', 'accept_new');
await Promise.all([p1, p2]);
// Three fetch calls for the three sync passes (dm, friend, profile) — not six.
expect(fetchCount).toBe(3);
});
it('swallows errors from syncPeerMutationLog so the handler does not throw', async () => {
const { onPeerActivated } = await import('./federationPeerActivation.js');
testDb.insert(schema.federationPeers).values({
id: 'peer-err', origin: 'https://peer-err.example', hmacSecret: 'secret',
status: 'active', lastSyncedAt: 0, createdAt: Date.now(),
}).run();
vi.spyOn(globalThis, 'fetch').mockImplementation(async () => {
throw new Error('network down');
});
await expect(onPeerActivated('peer-err', 'ensure_peered')).resolves.toBeUndefined();
});
});
@@ -0,0 +1,179 @@
import { getDb } from '../db/index.js';
import * as schema from '../db/schema.js';
import { and, eq } from 'drizzle-orm';
import { isFederationRelayEnabled } from './federationOutbox.js';
import { buildFederationHeaders, getOurOrigin } from './federationAuth.js';
import type { FederationRelayEvent } from '@backspace/shared';
export type PeerActivationReason =
| 'initiate_accepted'
| 'accept_rejected_override'
| 'accept_awaiting_approval'
| 'accept_pending'
| 'accept_new'
| 'approval_handshake'
| 'health_check_recovery'
| 'ensure_peered'
| 'startup_bootstrap';
// Dedup: concurrent activations for the same peerId share one promise.
const inFlightActivation = new Map<string, Promise<void>>();
/**
* Called whenever federation_peers.status transitions to 'active' for any reason.
* Two independent invariants — both run unconditionally:
* 1. Reset outbox backoff (nextRetryAt = now, attempts = 0) for this peer.
* 2. Pull-sync mutation log from peer's /api/federation/sync since lastSyncedAt.
*
* Call sites (must remain exhaustive — grep `onPeerActivated(` to audit):
* - routes/federation.ts /peer/initiate activation
* - routes/federation.ts /peer/accept existing-rejected override
* - routes/federation.ts /peer/accept existing-awaiting_approval
* - routes/federation.ts /peer/accept existing-pending
* - routes/federation.ts /peer/accept new-peer
* - routes/federation.ts /approval-requests/:id/approve
* - utils/federationWorker.ts health check recovery
* - utils/federationPeering.ts ensurePeered/performHandshake
* - utils/federationWorker.ts startup bootstrap (via startupBootstrapSync)
*
* Deduplicated by peerId — concurrent calls share one promise.
*/
export async function onPeerActivated(
peerId: string,
reason: PeerActivationReason,
): Promise<void> {
const existing = inFlightActivation.get(peerId);
if (existing) return existing;
const promise = (async () => {
try {
resetOutboxBackoff(peerId);
await syncPeerMutationLog(peerId, reason);
const { connectionManager } = await import('../ws/handler.js');
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
} catch (err) {
console.error(`[federation] onPeerActivated(${peerId}, ${reason}) failed:`, err);
}
})();
inFlightActivation.set(peerId, promise);
try {
await promise;
} finally {
inFlightActivation.delete(peerId);
}
}
/**
* Reset all outbox backoff state for a peer (nextRetryAt = now, attempts = 0).
* Unconditional across all entries of the peer — see spec §Invariant 1.
*/
export function resetOutboxBackoff(peerId: string): void {
const db = getDb();
const now = Date.now();
const result = db
.update(schema.federationOutbox)
.set({ nextRetryAt: now, attempts: 0 })
.where(eq(schema.federationOutbox.peerId, peerId))
.run();
if (result.changes > 0) {
console.log(`[federation] Reset backoff on ${result.changes} outbox entries for peer ${peerId}`);
}
}
/**
* Pull-sync mutation log from the peer's /api/federation/sync endpoint.
* Runs three contextType passes (dm, friend, profile), paginating each.
* Updates peer.lastSyncedAt to Date.now() on success; leaves it untouched
* on transient failure so the next activation retries.
*/
export async function syncPeerMutationLog(
peerId: string,
reason: PeerActivationReason,
): Promise<void> {
if (!isFederationRelayEnabled()) return;
const db = getDb();
const peer = db.select().from(schema.federationPeers)
.where(eq(schema.federationPeers.id, peerId)).get();
if (!peer || peer.status !== 'active') return;
const activePeer = peer; // narrowed by the guard above
const ourOrigin = getOurOrigin();
const signingSecret = (activePeer.pendingHmacSecret && activePeer.secretRotationAt)
? activePeer.pendingHmacSecret
: activePeer.hmacSecret;
console.log(`[federation] Sync-pull from ${activePeer.origin} (reason=${reason}, since=${activePeer.lastSyncedAt ?? 0})`);
let totalEvents = 0;
type SyncRequestBody = {
sinceTimestamp: number;
limit: number;
contextType?: 'friend' | 'profile';
};
async function runPass(contextType?: 'friend' | 'profile'): Promise<boolean> {
let since = activePeer.lastSyncedAt ?? 0;
while (true) {
const bodyObj: SyncRequestBody = { sinceTimestamp: since, limit: 100 };
if (contextType) bodyObj.contextType = contextType;
const body = JSON.stringify(bodyObj);
const headers = buildFederationHeaders(body, signingSecret, ourOrigin);
const resp = await fetch(`${activePeer.origin}/api/federation/sync`, {
method: 'POST', headers, body,
signal: AbortSignal.timeout(30_000),
});
if (!resp.ok) {
console.warn(`[federation] Sync-pull ${contextType ?? 'dm'} pass HTTP ${resp.status} for ${activePeer.origin}`);
return false;
}
const data = await resp.json() as { events: FederationRelayEvent[]; hasMore: boolean; checkpoint: number };
if (data.events.length === 0) return true;
const { processRelayEvents } = await import('../routes/federation.js');
await processRelayEvents(data.events, activePeer.origin, activePeer.origin, db);
totalEvents += data.events.length;
since = data.checkpoint;
if (!data.hasMore) return true;
}
}
try {
if (!(await runPass())) return;
if (!(await runPass('friend'))) return;
if (!(await runPass('profile'))) return;
db.update(schema.federationPeers)
.set({ lastSyncedAt: Date.now() })
.where(eq(schema.federationPeers.id, activePeer.id))
.run();
if (totalEvents > 0) {
console.log(`[federation] Sync-pull from ${activePeer.origin} replayed ${totalEvents} events`);
}
} catch (err) {
console.error(`[federation] Sync-pull from ${activePeer.origin} failed:`, err);
}
}
/**
* Startup bootstrap — scan for freshly-peered rows (status='active', lastSyncedAt=0)
* and run onPeerActivated for each. Replaces runInitialSyncForNewPeers.
* Invoked from startFederationWorkers.
*/
export async function startupBootstrapSync(): Promise<void> {
if (!isFederationRelayEnabled()) return;
const db = getDb();
const peers = db.select().from(schema.federationPeers)
.where(and(
eq(schema.federationPeers.status, 'active'),
eq(schema.federationPeers.lastSyncedAt, 0),
)).all();
for (const peer of peers) {
await onPeerActivated(peer.id, 'startup_bootstrap');
}
}
@@ -1,4 +1,4 @@
import { describe, it, expect, vi } from 'vitest'; import { describe, it, expect, vi, beforeEach } from 'vitest';
import type { EnsurePeeredResult } from './federationPeering.js'; import type { EnsurePeeredResult } from './federationPeering.js';
import { racePeering } from './federationPeering.js'; import { racePeering } from './federationPeering.js';
@@ -112,3 +112,59 @@ describe('racePeering', () => {
warnSpy.mockRestore(); warnSpy.mockRestore();
}); });
}); });
describe('ensurePeered needs_attention handling', () => {
beforeEach(() => {
vi.resetModules();
});
it('returns rejected without calling performHandshake when peer is in needs_attention', async () => {
const fakeDbGet = vi.fn().mockReturnValue({
id: 'peer-na',
origin: 'https://remote.example',
status: 'needs_attention',
hmacSecret: 'secret',
createdAt: Date.now(),
lastSyncedAt: 0,
});
vi.doMock('../db/index.js', () => ({
getDb: () => ({
select: () => ({
from: () => ({
where: () => ({
get: fakeDbGet,
}),
}),
}),
}),
}));
vi.doMock('../utils/federationAuth.js', () => ({
getOurOrigin: () => 'https://local.example',
generateHmacSecret: () => 'new-secret',
}));
vi.doMock('../routes/federation.js', () => ({
validateOrigin: (o: string) => o,
}));
vi.doMock('../utils/federationPeerActivation.js', () => ({
onPeerActivated: vi.fn(),
}));
const { ensurePeered } = await import('./federationPeering.js');
const fetchSpy = vi.spyOn(globalThis, 'fetch');
const result = await ensurePeered('https://remote.example');
expect(result.status).toBe('rejected');
if (result.status === 'rejected') {
expect(result.error).toContain('needs_attention');
}
// performHandshake must not have fired — no POST to /peer/accept
expect(fetchSpy).not.toHaveBeenCalled();
vi.restoreAllMocks();
});
});
@@ -4,6 +4,7 @@ import { eq } from 'drizzle-orm';
import { generateSnowflake } from './snowflake.js'; import { generateSnowflake } from './snowflake.js';
import { getOurOrigin, generateHmacSecret } from './federationAuth.js'; import { getOurOrigin, generateHmacSecret } from './federationAuth.js';
import { validateOrigin } from '../routes/federation.js'; import { validateOrigin } from '../routes/federation.js';
import { onPeerActivated } from './federationPeerActivation.js';
// ─── Types ─────────────────────────────────────────────────────────────────── // ─── Types ───────────────────────────────────────────────────────────────────
@@ -72,6 +73,9 @@ export async function ensurePeered(origin: string): Promise<EnsurePeeredResult>
// Unreachable peers were previously active — treat as active for peering // Unreachable peers were previously active — treat as active for peering
// (the health check will restore them; don't re-handshake) // (the health check will restore them; don't re-handshake)
return { status: 'active', peerId: existing.id }; return { status: 'active', peerId: existing.id };
case 'needs_attention':
// Admin intervention required — do not auto-heal via performHandshake
return { status: 'rejected', error: 'Peer in needs_attention — admin Reset required' };
case 'awaiting_approval': case 'awaiting_approval':
return { status: 'pending', error: 'Awaiting admin approval on remote instance' }; return { status: 'pending', error: 'Awaiting admin approval on remote instance' };
case 'pending': case 'pending':
@@ -158,6 +162,9 @@ async function performHandshake(
.run(); .run();
const { connectionManager } = await import('../ws/handler.js'); const { connectionManager } = await import('../ws/handler.js');
connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const }); connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
onPeerActivated(peerId, 'ensure_peered').catch(err =>
console.error('[federation] onPeerActivated from ensurePeered failed:', err)
);
return { status: 'active', peerId }; return { status: 'active', peerId };
} }
+19 -114
View File
@@ -2,7 +2,7 @@ import { getDb } from '../db/index.js';
import * as schema from '../db/schema.js'; import * as schema from '../db/schema.js';
import { eq, and, lte, asc, inArray, sql } from 'drizzle-orm'; import { eq, and, lte, asc, inArray, sql } from 'drizzle-orm';
import { config } from '../config.js'; import { config } from '../config.js';
import { isFederationRelayEnabled, queueOutboxEvent } from './federationOutbox.js'; import { isFederationRelayEnabled, queueOutboxEvent, appendMutationLog } from './federationOutbox.js';
import { runFederationJanitor } from './storageJanitor.js'; import { runFederationJanitor } from './storageJanitor.js';
import { buildFederationHeaders, getOurOrigin, generateHmacSecret, ROTATION_GRACE_PERIOD_MS } from './federationAuth.js'; import { buildFederationHeaders, getOurOrigin, generateHmacSecret, ROTATION_GRACE_PERIOD_MS } from './federationAuth.js';
import { evaluateAuthFailure, AUTH_FAILURE_THRESHOLD } from './federationAuthFailure.js'; import { evaluateAuthFailure, AUTH_FAILURE_THRESHOLD } from './federationAuthFailure.js';
@@ -10,8 +10,8 @@ import { generateSnowflake } from './snowflake.js';
import { getDmMessageWithUser } from '../routes/dm.js'; import { getDmMessageWithUser } from '../routes/dm.js';
import { connectionManager } from '../ws/handler.js'; import { connectionManager } from '../ws/handler.js';
import { generateThumbnail } from './thumbnail.js'; import { generateThumbnail } from './thumbnail.js';
import { processRelayEvents } from '../routes/federation.js';
import type { FederationRelayRequest, FederationRelayResponse, FederationRelayEvent } from '@backspace/shared'; import type { FederationRelayRequest, FederationRelayResponse, FederationRelayEvent } from '@backspace/shared';
import { onPeerActivated, startupBootstrapSync } from './federationPeerActivation.js';
import fs from 'node:fs'; import fs from 'node:fs';
import path from 'node:path'; import path from 'node:path';
import crypto from 'node:crypto'; import crypto from 'node:crypto';
@@ -703,6 +703,18 @@ function handleSizeRejection(
affectedUserIds, affectedUserIds,
}; };
appendMutationLog(
localMsg.sourceMessageId,
localMsg.dmChannelId,
'file_rejected',
JSON.stringify({
attachmentId: att?.id ?? entry.sourceUrl,
sourceFilename,
rejectionReason: 'size_limit_exceeded',
rejectionLimit: maxUploadSize,
affectedUserIds,
}),
);
queueOutboxEvent( queueOutboxEvent(
localMsg.sourceMessageId, localMsg.sourceMessageId,
localMsg.dmChannelId, localMsg.dmChannelId,
@@ -1054,6 +1066,8 @@ async function processHealthCheckTick(): Promise<void> {
console.log( console.log(
`[federation-worker] Peer ${peer.origin} recovered — marked active`, `[federation-worker] Peer ${peer.origin} recovered — marked active`,
); );
await onPeerActivated(peer.id, 'health_check_recovery');
} }
// If not ok, leave as unreachable — will check again next cycle // If not ok, leave as unreachable — will check again next cycle
} catch (err) { } catch (err) {
@@ -1076,124 +1090,15 @@ function scheduleJanitorTick(): void {
// ─── Lifecycle ────────────────────────────────────────────────────────────── // ─── Lifecycle ──────────────────────────────────────────────────────────────
/**
* Trigger checkpoint sync for peers that have never been synced (lastSyncedAt === 0).
* This catches historical messages that existed before the relay was enabled.
*/
async function runInitialSyncForNewPeers(): Promise<void> {
if (!isFederationRelayEnabled()) return;
const db = getDb();
const unsyncedPeers = db
.select()
.from(schema.federationPeers)
.where(and(
eq(schema.federationPeers.status, 'active'),
eq(schema.federationPeers.lastSyncedAt, 0),
))
.all();
if (unsyncedPeers.length === 0) return;
const ourOrigin = getOurOrigin();
for (const peer of unsyncedPeers) {
const signingSecret = (peer.pendingHmacSecret && peer.secretRotationAt)
? peer.pendingHmacSecret
: peer.hmacSecret;
try {
console.log(`[federation-worker] Running initial sync with ${peer.origin}...`);
let sinceTimestamp = 0;
let totalEvents = 0;
// Paginate through all events from the peer
while (true) {
const body = JSON.stringify({ sinceTimestamp, limit: 100 });
const headers = buildFederationHeaders(body, signingSecret, ourOrigin);
const response = await fetch(`${peer.origin}/api/federation/sync`, {
method: 'POST',
headers,
body,
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
console.error(`[federation-worker] Sync with ${peer.origin} failed: ${response.status}`);
break;
}
const data = await response.json() as { events: FederationRelayEvent[]; hasMore: boolean; checkpoint: number };
if (data.events.length === 0) break;
// Process events directly — no HTTP round-trip (FED-005)
await processRelayEvents(data.events, peer.origin, peer.origin, db);
totalEvents += data.events.length;
sinceTimestamp = data.checkpoint;
if (!data.hasMore) break;
}
// Second pass: sync friend events
let friendSinceTimestamp = 0;
while (true) {
const friendBody = JSON.stringify({ sinceTimestamp: friendSinceTimestamp, contextType: 'friend', limit: 100 });
const friendHeaders = buildFederationHeaders(friendBody, signingSecret, ourOrigin);
const friendResponse = await fetch(`${peer.origin}/api/federation/sync`, {
method: 'POST',
headers: friendHeaders,
body: friendBody,
signal: AbortSignal.timeout(30_000),
});
if (!friendResponse.ok) {
console.error(`[federation-worker] Friend sync with ${peer.origin} failed: ${friendResponse.status}`);
break;
}
const friendData = await friendResponse.json() as { events: FederationRelayEvent[]; hasMore: boolean; checkpoint: number };
if (friendData.events.length === 0) break;
// Process events directly — no HTTP round-trip (FED-005)
await processRelayEvents(friendData.events, peer.origin, peer.origin, db);
totalEvents += friendData.events.length;
friendSinceTimestamp = friendData.checkpoint;
if (!friendData.hasMore) break;
}
// Update lastSyncedAt so this doesn't run again
db.update(schema.federationPeers)
.set({ lastSyncedAt: Date.now() })
.where(eq(schema.federationPeers.id, peer.id))
.run();
if (totalEvents > 0) {
console.log(`[federation-worker] Initial sync with ${peer.origin}: ${totalEvents} events synced`);
} else {
console.log(`[federation-worker] Initial sync with ${peer.origin}: no events to sync`);
}
} catch (err) {
console.error(`[federation-worker] Initial sync with ${peer.origin} failed:`, err);
// Don't update lastSyncedAt — will retry next startup
}
}
}
export function startFederationWorkers(): void { export function startFederationWorkers(): void {
console.log('[federation-worker] Federation workers started'); console.log('[federation-worker] Federation workers started');
scheduleOutboxTick(); scheduleOutboxTick();
scheduleFileQueueTick(); scheduleFileQueueTick();
scheduleHealthCheckTick(); scheduleHealthCheckTick();
scheduleJanitorTick(); scheduleJanitorTick();
// Run initial sync for newly peered instances (async, non-blocking) // Bootstrap sync for freshly-peered rows (async, non-blocking)
runInitialSyncForNewPeers().catch((err) => { startupBootstrapSync().catch((err) => {
console.error('[federation-worker] Initial sync error:', err); console.error('[federation-worker] Startup bootstrap sync error:', err);
}); });
} }