docs(federation): document instance-epoch self-healing (Phase 2)
This commit is contained in:
Jannis Braun
@@ -534,6 +534,10 @@ Manages: federation peers list, pending approval requests (inbound + outbound),
|
||||
- Revoke calls `api.federation.revokePeer(peerId)` and removes from local list.
|
||||
- Peers in `needs_attention` status render with a rose "Needs Attention" pill and a single "Reset Peering" action. The action opens a danger-variant ConfirmDialog explaining that reset deletes the local peer record (cascade-removes outbox entries) and requires out-of-band re-peering with the remote admin.
|
||||
|
||||
- **Reset cleanup section** (`ResetCleanup`, instance-epoch self-healing §6.4) — the highest-priority attention surface, rendered above the peer list; returns `null` when there is nothing to clean up. Fetches `api.federation.peers()` + `api.federation.resetEvents()` and subscribes to `onFederationPeerResetDetected` (the `federation_peer_reset_detected` WS event) to refetch live. Two stacked surfaces:
|
||||
- **Reset-detected banner** — one persistent accent-rose banner per peer with `status === 'needs_attention' && needsAttentionReason === 'peer_reset_detected'`, distinguishing a wiped-and-reinstalled peer from a generic auth-failure peer. Its **Re-peer** button runs the existing one-click flow in order: `api.federation.resetPeer(id)` **then** `api.federation.initiatePeering({ remoteOrigin })` — resetting the stale local record *before* the fresh handshake so activation heals stale friendships/DMs against the new incarnation (warning-variant ConfirmDialog).
|
||||
- **Orphaned-accounts list** (from `GET /reset-events`, per origin with `orphanedAccounts.length > 0`) — each real account frozen by the reset quarantine, shown with its owned-spaces / membership / message counts. Per-row actions: **Keep** (default no-op resting/frozen state — the account stays `federationHomeOrphaned = 1`) and **Remove** (danger ConfirmDialog → `api.admin.deleteUser(id)`, i.e. the existing `DELETE /api/admin/users/:id` full purge). A Remove on a space-owning account returns the existing `409 { ownedSpaces }`; the UI surfaces a "transfer ownership first" toast rather than deleting.
|
||||
|
||||
#### StoragePanel
|
||||
|
||||
Manages: storage overview, file type breakdown, upload limit, orphan cleanup, media retention cleanup.
|
||||
|
||||
+27
-1
@@ -13,6 +13,8 @@ GET /auth/check-invite ?token= → CheckInviteResponse
|
||||
POST /auth/login { username, password } → { token, user }
|
||||
```
|
||||
|
||||
**`POST /auth/login`** — request/response shape unchanged, but two internal controls from instance-epoch self-healing gate the flow: (1) an account with `federationHomeOrphaned = 1` (home instance factory-reset) is rejected with the generic 401 *before* password verification; (2) the federated password self-heal now runs an **epoch guard** — it re-hashes the stale local password only if the home instance's authenticated epoch (`fetchPeerEpoch`) matches the trusted baseline, failing closed when the epoch differs or can't be determined. No wire-shape change. See `auth.md` §4.
|
||||
|
||||
**`POST /auth/register` gating** — branches on whether `homeInstance` is set:
|
||||
- **Federated path** (`homeInstance` set): gated solely by `instance_settings.federatedRegistrationOpen`. `inviteToken` is ignored entirely (not validated, not consumed). 403 `Federated registration is closed on this instance` when closed. Existing federated stubs (relay-created, `passwordHash = '!federation-replicated'`) upgrade in place — login is never blocked by this gate.
|
||||
- **Local path** (no `homeInstance`):
|
||||
@@ -321,7 +323,8 @@ type InviteRedemption = {
|
||||
```
|
||||
POST /federation/peer/initiate (admin) { remoteOrigin } → peer created
|
||||
POST /federation/peer/accept (public, IP rate-limited 10/min) { sourceOrigin, challenge, hmacSecret, instanceName?, instanceId?, approvalToken? } → { accepted, instanceName, instanceId } (200) | queued (202 + { approvalToken })
|
||||
GET /federation/peers (admin) → { peers[] } (no secrets)
|
||||
GET /federation/peers (admin) → { peers[] } (no secrets; each peer carries needsAttentionReason)
|
||||
GET /federation/reset-events (admin) → FederationResetEventsResponse
|
||||
DELETE /federation/peers/:id (admin) → { success } + outbox cleanup
|
||||
POST /federation/relay (HMAC-signed S2S) FederationRelayRequest (+ sourceInstanceId?) → { accepted[], rejected[] }
|
||||
POST /federation/sync (HMAC-signed S2S) { sinceTimestamp, limit?, dmChannelId?, federatedId?, contextType? } → { events[], hasMore, checkpoint }
|
||||
@@ -333,6 +336,29 @@ POST /federation/epoch (HMAC-signed S2S, HMAC-signed response) {}
|
||||
|
||||
**Handshake epoch exchange.** The handshake carries the **instance epoch** bidirectionally, mirroring `instanceName`: the request body's `instanceId` is the initiator's epoch (written to `federation_peers.peer_instance_id` on every authenticated activation path), and the 200 response body's `instanceId` is the responder's epoch (persisted by the initiator alongside `status='active'`). Older peers omit the field; the column stays `null` until the epoch-refresh/relay backstop fills it. Both are authenticated baselines — never overwritten by the unauthenticated `/instance/info` probe. **`FederationRelayRequest.sourceInstanceId`** stamps the sender's current epoch on every relay; because the whole body is HMAC-verified, a valid relay authentically carries the sender's incarnation id and populates `peer_instance_id` when null (fast-path baseline). See `federation.md` "Instance Epoch".
|
||||
|
||||
**`GET /api/federation/reset-events`** — admin-only, read-only. Backs the "Reset cleanup" admin surface (instance-epoch self-healing §6.4). Returns the durable `federation_reset_events` journal, each row augmented with the origin's current orphaned real accounts (`federationHomeOrphaned = 1`) for disposition:
|
||||
|
||||
```typescript
|
||||
type FederationOrphanedAccount = {
|
||||
id: string;
|
||||
username: string; // '!orphaned:{uid}@domain' for freed handles; real for space owners
|
||||
displayName: string | null;
|
||||
avatarColor: string | null;
|
||||
ownedSpaces: { id: string; name: string }[];
|
||||
spaceMemberCount: number; // # spaces the account is a member of
|
||||
messageCount: number; // # space messages authored
|
||||
};
|
||||
type FederationResetEvent = {
|
||||
origin: string; deadEpoch: string; newEpoch: string | null;
|
||||
detectedAt: number; resolvedAt: number | null;
|
||||
stubCount: number; orphanedAccountCount: number;
|
||||
orphanedAccounts: FederationOrphanedAccount[];
|
||||
};
|
||||
type FederationResetEventsResponse = { events: FederationResetEvent[] };
|
||||
```
|
||||
|
||||
Disposition actions reuse existing endpoints (no new mutating routes): one-click Re-peer = `POST /peers/:id/reset` → `POST /peer/initiate`; full-purge Remove = `DELETE /api/admin/users/:id` (owns-spaces → transfer first). **`needsAttentionReason`** (`'auth_failures' | 'peer_reset_detected' | null`) is now included on each `GET /federation/peers` peer object so the client can raise the persistent Reset-cleanup banner only for reset-detected peers. See `federation.md` "Instance Epoch" and `client-federation.md` §8.
|
||||
|
||||
**`POST /api/federation/users/lookup`** — HMAC-authenticated S2S endpoint. Resolves a username on this instance to its canonical `(homeUserId, profile snapshot)`. Used by the cross-instance friend-add flow on the sender's home server before queuing a `friend_request_create` event. Responds to native, non-deleted users only; ignores `discoverable`. Returns `{ found: false, code: 'user_not_found' }` for stubs, tombstoned users, or unknown handles. See `federation.md` §1 "S2S User Lookup" for the full contract.
|
||||
|
||||
**`POST /api/federation/epoch`** — HMAC-authenticated S2S endpoint returning this instance's persistent epoch (`{ instanceId }`). The **request** is HMAC-signed (only a peer holding the shared secret may call it; unknown/revoked peers → 403, bad signature → 401, missing headers → 400) **and the response body is HMAC-signed** with the same secret (`X-Federation-Signature/Timestamp/Nonce` response headers), so the caller can verify the epoch before writing it as the peer's trusted baseline (`federation_peers.peer_instance_id`). The value is already public via `/instance/info`; signing is for baseline-integrity, not confidentiality. Caller: `fetchPeerEpoch(peer)` (`utils/federationEpoch.ts`), which fails safe — 404 (not-yet-upgraded peer), bad/absent response signature, or network/timeout all return `null` (retry next tick). Populates the epoch baseline deterministically via the bounded periodic epoch-refresh. See `federation.md` "Instance Epoch" §3.2.
|
||||
|
||||
+17
-5
@@ -264,10 +264,11 @@ Validates format (same rules as local registration: 3-32 chars, `/^[a-z0-9_]+$/`
|
||||
2. Look up user by `username` (trimmed, lowercased)
|
||||
3. Reject if not found (generic "Invalid username or password")
|
||||
4. Reject if `isDeleted === 1` ("This account has been deleted")
|
||||
5. Verify password via bcrypt
|
||||
6. **If password invalid AND user is federated:** attempt self-healing (see below)
|
||||
7. **If password invalid AND user is local:** reject
|
||||
8. Sign JWT, return `{ token, user }`. **Note:** Login does NOT mutate `users.status`. A successful login does not by itself imply a live connection (the client may never establish a WebSocket due to network failure, mobile background, error path); writing `'online'` here would produce a permanently stuck-online row that no disconnect timer cleans up. The WebSocket auth path (`ws/handler.ts`) is the single source of truth for `status = 'online'`. See `docs/systems/activity-presence.md` "Boot Reset" for the mitigation that runs on server start.
|
||||
5. **Reject if `federationHomeOrphaned === 1`** (generic "Invalid username or password") — *before* password verification. This freezes any federated account whose home instance was factory-reset (a new incarnation stood up on the same domain), set by the reset quarantine (§6.3 below / `federation.md` "Instance Epoch"). Because it returns first, it blocks both the local-password path AND the self-heal path — nobody, including a new same-name user on the reset home, can authenticate into the dead-incarnation account. Reversible (admin Keep/Remove, or the real user re-registers into a fresh account).
|
||||
6. Verify password via bcrypt
|
||||
7. **If password invalid AND user is federated:** attempt self-healing (see below)
|
||||
8. **If password invalid AND user is local:** reject
|
||||
9. Sign JWT, return `{ token, user }`. **Note:** Login does NOT mutate `users.status`. A successful login does not by itself imply a live connection (the client may never establish a WebSocket due to network failure, mobile background, error path); writing `'online'` here would produce a permanently stuck-online row that no disconnect timer cleans up. The WebSocket auth path (`ws/handler.ts`) is the single source of truth for `status = 'online'`. See `docs/systems/activity-presence.md` "Boot Reset" for the mitigation that runs on server start.
|
||||
|
||||
### Federation Password Self-Healing
|
||||
|
||||
@@ -276,7 +277,7 @@ When local bcrypt verification fails for a user with `homeInstance` set:
|
||||
1. Extract base username (strip `@domain` if present)
|
||||
2. POST to `https://{homeInstance}/api/auth/login` with base username and provided password
|
||||
3. Timeout: 10 seconds (`AbortController`)
|
||||
4. **If home instance accepts (200):**
|
||||
4. **If home instance accepts (200):** run the **epoch guard** (see below) before re-hashing. If the guard passes:
|
||||
- Re-hash password locally: `hashPassword(password)`
|
||||
- Update local `passwordHash` -- but **do NOT set `passwordChangedAt`** (this is a state correction, not a password change; setting it would invalidate existing valid JWTs on this instance)
|
||||
- Log the self-healing event
|
||||
@@ -286,6 +287,17 @@ When local bcrypt verification fails for a user with `homeInstance` set:
|
||||
|
||||
This flow ensures that when a federated user changes their password on their home instance, they can still log in on remote instances even if the remote's hash is stale.
|
||||
|
||||
#### Epoch guard (instance-epoch self-healing §6.3a)
|
||||
|
||||
Before re-hashing, the self-heal confirms the home instance is the **same incarnation** the trusted baseline was established with. Without this, a factory-reset home accepting a *new* same-name user's password would silently hand that stranger the established account. The guard does **not** trust the login-response body (TLS-MITM-bypassable, and a reset home would just echo its new epoch); it **reuses the authenticated `fetchPeerEpoch(peer)`** (`utils/federationEpoch.ts`, HMAC-signed request *and* response) to read the home's current epoch, then compares it to `federation_peers.peerInstanceId` for that origin (three-way):
|
||||
|
||||
- **No peer row / `peerInstanceId` is null** (legacy/never-tracked): allow — fall through to self-heal (no regression).
|
||||
- **Baseline on record AND `fetchPeerEpoch` returns a *different* epoch:** refuse self-heal → "Invalid username or password" (the hijack case).
|
||||
- **Baseline on record AND `fetchPeerEpoch` returns `null`** — epoch cannot be determined (peer too old → 404, unreachable, bad/absent response signature, **or the reset peer's desynced secret rejecting our signed request**): **fail closed — refuse self-heal.**
|
||||
- **Baseline on record AND the epoch matches:** allow.
|
||||
|
||||
Trade-off: the separate authenticated call can fail independently of the login POST, so a transient home outage during a legitimate stale-hash login fails closed. This is security-over-availability on a rare, recoverable path (fallback: a normal password change once the home is reachable); trusting an unauthenticated body would re-open the hijack. A reset peer's `null` result also doubles as a reset signal — the guard is correct even before reset-detection has flagged the peer. The direct-login freeze (step 5 of the Login Flow, `federationHomeOrphaned = 1`) is the post-re-peer complement: once the admin re-peers and the baseline updates to the new epoch, the epoch guard alone would pass again, so the freeze is what keeps the dead-incarnation account locked.
|
||||
|
||||
---
|
||||
|
||||
## 5. Password Change
|
||||
|
||||
@@ -475,6 +475,13 @@ A new `federationStore.ts` slice (separate from `instanceStore`) holds:
|
||||
|
||||
This slice is intentionally separate from `instanceStore` because the data is per-user (not per-instance) and lives on the home server only. WebSocket handlers route `peering_subscription_changed` and `peering_notification_received` events into this slice's refetch actions.
|
||||
|
||||
### Reset cleanup (instance-epoch self-healing §6.4)
|
||||
|
||||
Modeled on the peering-approval surface above, the FederationPanel's `ResetCleanup` component (`admin.md` "FederationPanel") is the admin surface for a factory-reset peer. It fetches `api.federation.peers()` + `api.federation.resetEvents()` (`GET /api/federation/reset-events`) and subscribes to `onFederationPeerResetDetected(cb)` — the client handler for the `federation_peer_reset_detected` admin WS event (`useWebSocket.ts`) — to refetch live. Two surfaces:
|
||||
|
||||
- **Reset-detected banner** — one persistent accent-rose banner per peer with `status === 'needs_attention' && needsAttentionReason === 'peer_reset_detected'` (the new `needsAttentionReason` field distinguishes a reset from a generic auth-failure). **Re-peer** runs `resetPeer(id)` **then** `initiatePeering({ remoteOrigin })` — reset-before-handshake so activation heals the stale graph against the new incarnation.
|
||||
- **Orphaned-accounts list** — real accounts frozen by the server-side reset quarantine (`FederationOrphanedAccount`: owned-spaces / membership / message counts). **Keep** is the no-op frozen resting state; **Remove** reuses `api.admin.deleteUser(id)` (`DELETE /api/admin/users/:id`, full purge). A Remove on a space owner surfaces the existing `409 { ownedSpaces }` as a "transfer ownership first" toast instead of deleting.
|
||||
|
||||
---
|
||||
|
||||
## 9. Relationship to S2S Federation
|
||||
|
||||
@@ -34,7 +34,7 @@ IDs: Snowflake text, permissions: bigint decimal strings
|
||||
| showActivity | integer NOT NULL | 1 | Rich presence visibility |
|
||||
| federationRegistryUpdatedAt | integer | 0 | LWW timestamp for federation registry sync |
|
||||
| federationHealPending | integer | 0 | Instance-epoch self-healing: set when a replicated identity is flagged for re-heal after a peer reset |
|
||||
| federationHomeOrphaned | integer | 0 | Instance-epoch self-healing: set when this user's home instance was factory-reset and the account could not be re-linked |
|
||||
| federationHomeOrphaned | integer | 0 | Instance-epoch self-healing (now live, Phase 2): **set** to 1 by `quarantineOrphanedAccounts` on every real account whose home instance was factory-reset (freeze); **read** by the login flow (rejected before password verify — `auth.md` §4) and the `GET /api/federation/reset-events` admin surface. Reversible via admin Keep/Remove |
|
||||
| createdAt | integer NOT NULL | | Epoch ms |
|
||||
|
||||
### spaces
|
||||
|
||||
@@ -357,7 +357,18 @@ Detection (`markPeerReset`) only snapshots + journals + notifies; it destroys no
|
||||
|
||||
**Soft-tombstone (pure stubs only).** For every user that is `federation_heal_pending = 1` AND `password_hash = '!federation-replicated'` (pure S2S stub sentinel) AND matches the origin (`homeInstanceMatch`), calls `tombstoneUser(uid, { purgeContent: false })`. The `purgeContent: false` is **non-negotiable** — the default (`true`) irreversibly deletes this box's reactions and authored space messages, violating the invariant that a remote's reset never destroys our non-re-syncable content. The soft tombstone clears exactly the relationship rows that cause the bug (`friends`, `friend_requests`, `dm_members`, …) so stale friendships/DMs clear and re-adds work. Flags are then cleared **keyed by the stub id list** (not by re-querying the sentinel — `tombstoneUser` has already randomized `password_hash`).
|
||||
|
||||
**Real federated accounts left intact.** A flagged user that is NOT a stub (`password_hash != '!federation-replicated'`) carries real, non-re-syncable local content. It is **never** auto-tombstoned — it stays `federation_heal_pending = 1` and fully intact for the Phase 2 quarantine/admin surface (design §6.3).
|
||||
**Real federated accounts — quarantine (design §6.3b).** A flagged user that is NOT a stub (`password_hash != '!federation-replicated'`) carries real, non-re-syncable local content and is **never** auto-tombstoned. In the genuine-reset branch, after the stub soft-tombstone loop, `healResetIncarnation` calls `quarantineOrphanedAccounts(origin)`:
|
||||
|
||||
- For every flagged real account (`federation_heal_pending = 1`, non-stub, `isDeleted = 0`, `homeInstanceMatch`): set `federation_home_orphaned = 1` (**freeze**) and clear `federation_heal_pending`.
|
||||
- **The freeze is universal** — applied even to space owners. It is the barrier that closes the *post*-re-peer hijack: once re-peered, the trusted baseline updates to the new epoch, so the login epoch guard reads "match" again and no longer blocks a same-name hijack; only the freeze does. Enforcement is in `auth.ts` (direct login rejects `federation_home_orphaned = 1` before password verify — see `auth.md` §4).
|
||||
- **The rename is conditional.** A non-owner is renamed `username → '!orphaned:{uid}@{domain}'` to **free the handle**, so a returning same-name user re-registers into a clean fresh account instead of colliding. This defends BOTH the login uniqueness check AND the registration tier-2 stub-resolution upgrade (`findFederatedUser`). A space owner is **not** renamed (renaming an `ownerId`-referenced account orphans the reference) — it stays frozen and is surfaced to the admin to resolve ownership by hand.
|
||||
- Content (space messages, memberships, reactions) is preserved in all cases. The returned count refreshes the journal's `orphaned_account_count`.
|
||||
|
||||
**Login self-heal epoch guard (design §6.3a).** The federated password self-heal (`auth.ts` §4) now gates re-hashing on the home instance's current epoch, read via the authenticated `fetchPeerEpoch(peer)` (HMAC-signed both ways): no baseline on record → allow (legacy); baseline differs from the fetched epoch → refuse; epoch can't be determined (`fetchPeerEpoch` null — 404/unreachable/bad-sig/desynced secret) → **fail closed/refuse**; match → allow. Closes the *pre*-re-peer hijack (a reset home accepting a new same-name user's password); the universal quarantine freeze closes the post-re-peer window. Full three-way in `auth.md` §4.
|
||||
|
||||
**Reset-events admin surface (`GET /api/federation/reset-events`).** Admin-only, read-only. Returns the durable `federation_reset_events` journal joined with each origin's current orphaned real accounts (`federation_home_orphaned = 1`, `homeInstanceMatch`), each with `ownedSpaces`, `spaceMemberCount`, and authored-`messageCount` for disposition. Response type `FederationResetEventsResponse` (`{ events: FederationResetEvent[] }`, each event carrying `orphanedAccounts: FederationOrphanedAccount[]`). Disposition actions reuse existing endpoints — one-click Re-peer (`/peers/:id/reset` → `/peer/initiate`) and full-purge Remove (`DELETE /api/admin/users/:id`, owns-spaces → transfer first). See `admin.md` "FederationPanel" and `client-federation.md` §8.
|
||||
|
||||
**`needsAttentionReason` on the peer API.** `GET /api/federation/peers` returns `needsAttentionReason: 'auth_failures' | 'peer_reset_detected' | null` per peer, so the admin UI distinguishes a reset-detected peer (persistent Reset-cleanup banner + one-click Re-peer) from a generic auth-failure peer (plain "Reset Peering").
|
||||
|
||||
### S2S Identity Deletion (`DELETE /api/federation/identity`)
|
||||
|
||||
|
||||
@@ -194,7 +194,7 @@ reason: `'displaced'` (new tab) | `'session_closed'`
|
||||
|------|--------|-------|
|
||||
| `federation_file_rejected` | messageId, dmChannelId, attachmentId, affectedUsers[] | DM members |
|
||||
| `federation_approval_request_received` | — (refetch trigger; payload: `{ type }`) | admins. Fires for **both** inbound peering requests (remote → us) AND outbound queue creation when the [Outbound Peering Gate](federation.md#outbound-peering-gate) creates a `peer_approval_requests` row in response to a user_action. Payload shape unchanged from the inbound-only behavior; only the firing surface widened. |
|
||||
| `federation_peer_reset_detected` | `{ origin: string }` | admins. Fires from `markPeerReset` when a peer's advertised instance epoch differs from the trusted baseline (a wipe-and-reinstall on the same domain — see [Reset Detection](federation.md#reset-detection-markpeerreset--utilsfederationresetts)). Detection-only: the peer was routed to `needs_attention` (reason `peer_reset_detected`) with no rekey/tombstone. Paired with a `federation_peers_changed` broadcast; client surfaces the reset for one-click Re-peer. |
|
||||
| `federation_peer_reset_detected` | `{ origin: string }` | admins. Fires from `markPeerReset` when a peer's advertised instance epoch differs from the trusted baseline (a wipe-and-reinstall on the same domain — see [Reset Detection](federation.md#reset-detection-markpeerreset--utilsfederationresetts)). Detection-only: the peer was routed to `needs_attention` (reason `peer_reset_detected`) with no rekey/tombstone. Paired with a `federation_peers_changed` broadcast. **Client handler:** `onFederationPeerResetDetected(cb)` (`useWebSocket.ts`) — the FederationPanel's Reset-cleanup surface subscribes and refetches `GET /api/federation/peers` + `GET /api/federation/reset-events`, raising a persistent banner with one-click Re-peer (see `admin.md` "FederationPanel", `client-federation.md` §8). |
|
||||
| `peering_subscription_changed` | — (refetch trigger; payload: `{ type }`) | the subscribing user (all of their connected sessions). Fires when a `peer_approval_subscribers` row belonging to the user is created, modified, or deleted (gate fan-in, user cancel, parent cascade). Client refetches `GET /api/federation/peering-subscriptions`. |
|
||||
| `peering_notification_received` | `{ type, kind: 'approved' \| 'denied' \| 'expired' }` | the user the notification belongs to. Fires when a `peer_approval_notifications` row is created (`onPeerActivated` outbound fanout, outbound `/deny` fanout, janitor outbound expiry). Client refetches `GET /api/federation/peering-notifications` and may surface a transient toast for online users. |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user