feat(federation): peer_reset_pending guard during limbo window

This commit is contained in:
Jannis Braun
2026-07-01 22:32:40 +02:00
parent 7ef1ded116
commit 7d8c9c9d8d
6 changed files with 334 additions and 1 deletions
+9
View File
@@ -333,6 +333,15 @@ In a single transaction, `markPeerReset`:
Legacy peers advertise no epoch (`instanceId` null), so detection requires a non-null observed epoch differing from a non-null stored baseline — legacy peers never trigger it, and the existing `auth_failures → needs_attention → manual Reset` path continues unchanged for them.
### Limbo-window user error (`peer_reset_pending`)
Between reset detection (`markPeerReset`) and the admin's one-click Re-peer, the stale identity graph still exists and no heal has run (design §5.3). During this window a user re-adding a same-name friend, or creating a DM to that origin, would otherwise hit a confusing `already_friends` (stale friendship bound to the dead incarnation) or `peer_rejected` (the peer now sits in `needs_attention`, tripping `ensurePeered`). Both user-facing hot paths short-circuit with a clearer **409 `{ error: 'peer_reset_pending' }`**:
- **Friend-add** (`social.ts` `POST /api/social/requests`, federated branch): after `resolveOriginFromHostname` yields `peerOrigin` and **before** the peering/lookup/`already_friends` checks.
- **DM-create** (`dm.ts` `POST /api/dm`, `homeUserId + homeInstance` branch): before stub creation, so no un-flagged stub is left behind. The target origin is resolved with `resolveOriginFromHostname(new URL(canonicalizeHomeInstance(homeInstance)).host)`.
Both perform an **O(1) point lookup** on the `federation_reset_events` origin PRIMARY KEY (`origin = peerOrigin AND resolved_at IS NULL`). `peerOrigin` (from `resolveOriginFromHostname`, which returns the stored `federation_peers.origin` verbatim) is exactly the string `markPeerReset` journals, so the query is a single indexed hit/miss. The guard **only** short-circuits when an unresolved row exists; the common case — no reset in progress — is one indexed miss and the normal path proceeds byte-for-byte unchanged. Once the admin re-peers and `healResetIncarnation` resolves the journal (`resolved_at` set), the guard stops firing and the freshly-clean graph accepts the re-add.
### Data Self-Heal (`healResetIncarnation` — `utils/federationReset.ts`)
Detection (`markPeerReset`) only snapshots + journals + notifies; it destroys nothing. The actual heal is `healResetIncarnation(origin, newEpoch, reason)`, fired from `onPeerActivated` (`utils/federationPeerActivation.ts`) **after an admin-authenticated re-peer**, keyed to the confirmed epoch change (design §6). It runs **before** the mutation-log re-sync in `onPeerActivated` so re-sync repopulates onto a clean slate, and it runs **outside any transaction** (`tombstoneUser` opens its own; better-sqlite3 throws on a nested `BEGIN`).
+1
View File
@@ -293,6 +293,7 @@ As of 2026-04-25, the sender's home server owns the entire federated friend-add
1. **Parse target.** If `body.username` contains no `@`, or the domain after `@` normalizes to this server's own host, fall through to the local-only path (unchanged).
2. **resolveOriginFromHostname(targetDomain)** — resolves the target peer's full origin URL. Prefers a stored `federation_peers` row matching the typed host; falls back to mirroring `getOurOrigin()`'s scheme. Returns null → 400 `invalid_target_domain`.
2a. **Limbo-window guard → 409 `peer_reset_pending`.** O(1) point lookup on the `federation_reset_events` origin PRIMARY KEY: if an **unresolved** row exists for `peerOrigin` (`origin = peerOrigin AND resolved_at IS NULL`), the peer was reset-detected (wipe-and-reinstall) but the admin has not yet re-peered — the local friendship/stub graph is still bound to the dead incarnation. Return 409 `peer_reset_pending` instead of the confusing `already_friends` (stale friendship) or `peer_rejected` (the `needs_attention` peer would otherwise trip `ensurePeered`). `peerOrigin` is the exact string `markPeerReset` journals (the peer's `federation_peers.origin`), so the match is a single indexed lookup; no reset in progress → one indexed miss → the normal path proceeds unchanged. See `docs/systems/federation.md` (instance-epoch self-healing) and the design spec §5.3. The equivalent guard runs on federated DM-create (`POST /api/dm`, `dm.ts`).
3. **Authority defense.** If the calling user's `homeInstance` is set and does not normalize to this server's own host (checked via `normalizeOriginForCompare`), return 403 `not_authoritative_for_sender`. Prevents replicated/federated users from queueing relay events the home server isn't authoritative for. Runs before peering to fail fast.
4. **ensurePeered(peerOrigin)** — blocks on the result. Status → HTTP mapping:
- `'active'` → continue
@@ -0,0 +1,165 @@
import { describe, it, expect, beforeEach, vi } from 'vitest';
import Fastify, { type FastifyInstance } from 'fastify';
import Database from 'better-sqlite3';
import { drizzle } from 'drizzle-orm/better-sqlite3';
import { eq } from 'drizzle-orm';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import * as schema from '../db/schema.js';
import { setWorkerId } from '../utils/snowflake.js';
setWorkerId(1);
const __dirname = path.dirname(fileURLToPath(import.meta.url));
type TestDb = ReturnType<typeof drizzle<typeof schema>>;
let sqlite: Database.Database;
let testDb: TestDb;
const currentUserId = 'user-A';
vi.mock('../db/index.js', () => ({
getDb: () => testDb,
schema,
}));
vi.mock('../utils/auth.js', () => ({
authenticate: async (req: { userId?: string }) => {
req.userId = currentUserId;
},
}));
vi.mock('../ws/handler.js', () => ({
connectionManager: {
sendToUser: vi.fn(),
sendToDmMembers: vi.fn(),
sendToAdmins: vi.fn(),
getAllOnlineUserIds: () => [],
},
}));
vi.mock('../utils/federationOutbox.js', async () => {
const actual = await vi.importActual<typeof import('../utils/federationOutbox.js')>('../utils/federationOutbox.js');
return {
...actual,
isFederationRelayEnabled: () => true,
queueDmCloseRelay: vi.fn(),
sendTypingRelay: vi.fn(),
queueDmRelay: vi.fn(),
queueOutboxEvent: vi.fn(),
appendMutationLog: vi.fn(),
};
});
function applyMigrations(db: Database.Database): void {
const migrationsDir = path.resolve(__dirname, '../../drizzle');
const files = fs.readdirSync(migrationsDir).filter(f => f.endsWith('.sql')).sort();
for (const f of files) {
const sqlText = fs.readFileSync(path.join(migrationsDir, f), 'utf8');
const statements = sqlText.split(/-->\s*statement-breakpoint/);
for (const stmt of statements) {
const clean = stmt.trim();
if (clean) db.exec(clean);
}
}
}
function seedCaller(): void {
testDb.insert(schema.users).values({
id: 'user-A',
username: 'alice',
displayName: 'Alice',
passwordHash: 'x',
homeUserId: 'user-A',
homeInstance: null,
createdAt: Date.now(),
}).run();
}
/** The reset peer's persistent row (still present during the limbo window — only
* deleted on admin Re-peer). Its `origin` is the exact string markPeerReset journals. */
function seedPeer(): void {
testDb.insert(schema.federationPeers).values({
id: 'peer-remote',
origin: 'https://remote.example',
hmacSecret: 'secret',
status: 'needs_attention',
needsAttentionReason: 'peer_reset_detected',
peerInstanceId: 'dead-epoch',
observedPeerInstanceId: 'new-epoch',
createdAt: Date.now(),
}).run();
}
function seedResetEvent(resolvedAt: number | null): void {
testDb.insert(schema.federationResetEvents).values({
origin: 'https://remote.example',
deadEpoch: 'dead-epoch',
newEpoch: resolvedAt === null ? null : 'new-epoch',
detectedAt: Date.now(),
resolvedAt,
stubCount: 1,
orphanedAccountCount: 0,
}).run();
}
async function buildApp(): Promise<FastifyInstance> {
const app = Fastify({ logger: false });
const { dmRoutes } = await import('./dm.js');
await app.register(dmRoutes);
await app.ready();
return app;
}
describe('POST /api/dm — limbo-window peer_reset_pending guard', () => {
beforeEach(() => {
sqlite = new Database(':memory:');
testDb = drizzle(sqlite, { schema });
applyMigrations(sqlite);
seedCaller();
seedPeer();
});
it('returns 409 peer_reset_pending when creating a federated DM to a reset-pending origin', async () => {
seedResetEvent(null);
const app = await buildApp();
const res = await app.inject({
method: 'POST',
url: '/api/dm',
payload: { homeUserId: 'remote-bob', homeInstance: 'https://remote.example' },
});
expect(res.statusCode).toBe(409);
expect(res.json().error).toBe('peer_reset_pending');
// No stub created and no DM channel created for the reset-pending peer.
expect(testDb.select().from(schema.dmChannels).all()).toHaveLength(0);
expect(testDb.select().from(schema.users).where(eq(schema.users.homeUserId, 'remote-bob')).all()).toHaveLength(0);
});
it('proceeds normally when the reset event is RESOLVED', async () => {
seedResetEvent(Date.now());
const app = await buildApp();
const res = await app.inject({
method: 'POST',
url: '/api/dm',
payload: { homeUserId: 'remote-bob', homeInstance: 'https://remote.example' },
});
expect(res.statusCode).toBe(201);
expect(res.json().federatedId).toMatch(/^[a-f0-9]{32}$/);
});
it('proceeds normally when NO reset event exists for the origin', async () => {
const app = await buildApp();
const res = await app.inject({
method: 'POST',
url: '/api/dm',
payload: { homeUserId: 'remote-bob', homeInstance: 'https://remote.example' },
});
expect(res.statusCode).toBe(201);
expect(res.json().federatedId).toMatch(/^[a-f0-9]{32}$/);
});
});
+35
View File
@@ -47,6 +47,7 @@ import {
normalizeIconForWire,
} from '../utils/federationOutbox.js';
import { getOurOrigin, canonicalizeHomeInstance } from '../utils/federationAuth.js';
import { resolveOriginFromHostname } from '../utils/federationOriginResolve.js';
import type { FederationRelayEvent } from '@backspace/shared';
import { resolveLocalUser, resolveOrCreateReplicatedUser } from './federation.js';
@@ -928,6 +929,40 @@ export async function dmRoutes(app: FastifyInstance): Promise<void> {
let targetUser: typeof schema.users.$inferSelect | undefined;
if (homeUserId && homeInstance) {
// Limbo-window guard (federation instance-epoch self-healing §5.3).
// If the target's home instance was reset-detected but the admin has not yet
// re-peered, an UNRESOLVED `federation_reset_events` row exists for its origin.
// Creating a DM now would bind to stale, dead-incarnation identity state, so
// surface a clear `peer_reset_pending` instead of silently forming a doomed
// channel. Checked BEFORE stub creation so no un-flagged stub is left behind.
//
// The journal is keyed by the peer's `federation_peers.origin` (the exact string
// `markPeerReset` stores). `resolveOriginFromHostname` returns that stored origin
// verbatim, giving an O(1) point lookup on the origin PRIMARY KEY; the common case
// (no reset) is a single indexed miss and the normal path proceeds unchanged.
const canon = canonicalizeHomeInstance(homeInstance);
let peerOrigin: string | null = null;
if (canon) {
try {
peerOrigin = resolveOriginFromHostname(new URL(canon).host);
} catch {
peerOrigin = null;
}
}
if (peerOrigin) {
const pendingReset = db
.select({ origin: schema.federationResetEvents.origin })
.from(schema.federationResetEvents)
.where(and(
eq(schema.federationResetEvents.origin, peerOrigin),
isNull(schema.federationResetEvents.resolvedAt),
))
.get();
if (pendingReset) {
return reply.code(409).send({ error: 'peer_reset_pending', statusCode: 409 });
}
}
// Federated identity: resolve or create a replicated user stub
targetUser = resolveOrCreateReplicatedUser(homeUserId, homeInstance, db) ?? undefined;
} else if (userId && typeof userId === 'string') {
@@ -457,3 +457,100 @@ describe('POST /api/social/requests — federated branch (authority + self-frien
expect(body.requestId).toBe('incoming-req');
});
});
describe('POST /api/social/requests — federated branch (limbo-window peer_reset_pending)', () => {
beforeEach(() => {
seedSelf();
resolveOriginFromHostnameMock.mockReturnValue('https://orbit.test');
});
function seedResetEvent(resolvedAt: number | null): void {
testDb.insert(schema.federationResetEvents).values({
origin: 'https://orbit.test',
deadEpoch: 'dead-epoch',
newEpoch: resolvedAt === null ? null : 'new-epoch',
detectedAt: Date.now(),
resolvedAt,
stubCount: 1,
orphanedAccountCount: 0,
}).run();
}
it('returns 409 peer_reset_pending when an UNRESOLVED reset event exists for the target origin', async () => {
seedResetEvent(null);
// Even a stale friendship must NOT surface as `already_friends` during the limbo window.
testDb.insert(schema.users).values({
id: 'stub-alice',
username: 'remote-alice@orbit.test',
displayName: 'Alice',
passwordHash: '!federation-replicated',
status: 'offline',
isAdmin: 0,
homeInstance: 'orbit.test',
homeUserId: 'remote-alice-old',
createdAt: Date.now(),
}).run();
testDb.insert(schema.friends).values({
userId: CALLER_ID,
friendId: 'stub-alice',
createdAt: Date.now(),
}).run();
const app = await buildApp();
const res = await app.inject({
method: 'POST',
url: '/api/social/requests',
payload: { username: 'alice@orbit.test' },
});
expect(res.statusCode).toBe(409);
expect(JSON.parse(res.body).error).toBe('peer_reset_pending');
// Short-circuits before peering/lookup — neither is consulted.
expect(ensurePeeredMock).not.toHaveBeenCalled();
expect(lookupRemoteUserMock).not.toHaveBeenCalled();
// No new request row created.
expect(testDb.select().from(schema.friendRequests).all()).toHaveLength(0);
});
it('proceeds normally when the reset event is RESOLVED (resolved_at set)', async () => {
seedResetEvent(Date.now());
ensurePeeredMock.mockResolvedValue({ status: 'active', peerId: 'peer-1' });
lookupRemoteUserMock.mockResolvedValue({
ok: true,
homeUserId: 'remote-alice',
username: 'alice',
profile: { displayName: 'Alice', avatar: null, avatarColor: 'mint', banner: null, bio: null },
});
const app = await buildApp();
const res = await app.inject({
method: 'POST',
url: '/api/social/requests',
payload: { username: 'alice@orbit.test' },
});
expect(res.statusCode).toBe(201);
expect(ensurePeeredMock).toHaveBeenCalled();
expect(JSON.parse(res.body).success).toBe(true);
});
it('proceeds normally when NO reset event exists for the origin', async () => {
ensurePeeredMock.mockResolvedValue({ status: 'active', peerId: 'peer-1' });
lookupRemoteUserMock.mockResolvedValue({
ok: true,
homeUserId: 'remote-alice',
username: 'alice',
profile: { displayName: 'Alice', avatar: null, avatarColor: 'mint', banner: null, bio: null },
});
const app = await buildApp();
const res = await app.inject({
method: 'POST',
url: '/api/social/requests',
payload: { username: 'alice@orbit.test' },
});
expect(res.statusCode).toBe(201);
expect(ensurePeeredMock).toHaveBeenCalled();
});
});
+27 -1
View File
@@ -1,5 +1,5 @@
import type { FastifyInstance, FastifyRequest, FastifyReply } from 'fastify';
import { eq, and, or, ne, like, sql, inArray } from 'drizzle-orm';
import { eq, and, or, ne, like, sql, inArray, isNull } from 'drizzle-orm';
import { getDb, getRawDb, schema } from '../db/index.js';
import { authenticate } from '../utils/auth.js';
import { generateSnowflake } from '../utils/snowflake.js';
@@ -183,6 +183,32 @@ async function handleFederatedFriendRequest(
return reply.code(400).send({ error: 'invalid_target_domain', statusCode: 400, domain: targetDomain });
}
// 1a. Limbo-window guard (federation instance-epoch self-healing §5.3).
// If this peer's home instance was reset-detected but the admin has not yet
// re-peered, an UNRESOLVED `federation_reset_events` row exists for its origin
// (the peer sits in `needs_attention`, its local friendship/stub graph still
// bound to the dead incarnation). Without this guard the re-add would surface
// a confusing `already_friends` (stale friendship) or `peer_rejected` (the
// needs_attention peer) — neither of which tells the user what to do. Return a
// clear `peer_reset_pending` instead.
//
// `resolveOriginFromHostname` returns the peer's stored `federation_peers.origin`
// verbatim, which is exactly the string `markPeerReset` journals as
// `federation_reset_events.origin` (its PRIMARY KEY), so this is an O(1) indexed
// point lookup. The common case — no reset in progress — is a single indexed miss
// and the normal path proceeds unchanged.
const pendingReset = db
.select({ origin: schema.federationResetEvents.origin })
.from(schema.federationResetEvents)
.where(and(
eq(schema.federationResetEvents.origin, peerOrigin),
isNull(schema.federationResetEvents.resolvedAt),
))
.get();
if (pendingReset) {
return reply.code(409).send({ error: 'peer_reset_pending', statusCode: 409 });
}
// 2. ensurePeered — block until 'active', or surface peer status as error
const peering = await ensurePeered(peerOrigin, {
kind: 'user_action',