feat(federation): near-instant reset detection — probe epoch at the auth-failure transition + on worker startup

A reset peer reaches needs_attention via the auth-failure path (HMAC desynced by
the new incarnation) without passing through unreachable, so the 5s recovery
probe never saw it — detection waited up to a full 15-min health-check cycle
before 'Re-peer & heal' surfaced. Extract detectResetForPeer() and fire it
event-driven at the transition, plus a startup sweep for already-stuck peers.
15-min tick remains the backstop.
This commit is contained in:
Jannis Braun
2026-07-02 20:10:35 +02:00
parent 13d050c1bb
commit 54ab660204
4 changed files with 138 additions and 10 deletions
+5 -1
View File
@@ -332,7 +332,11 @@ In a single transaction, `markPeerReset`:
**Detection sources (all three wired in this feature):**
- **Inbound handshake** — `/peer/accept` landing on an `active`/`needs_attention` row (`routes/federation.ts`): before the idempotent-200 return, `if (reqInstanceId && existing.peerInstanceId && reqInstanceId !== existing.peerInstanceId) markPeerReset(...)`. The guard still returns 200 and **still does not rekey** — the anti-hijack property is preserved verbatim; detection is layered on top.
- **Reachability probe (`unreachable` peers)** — `probePeerReachable()` (`utils/federationRecovery.ts`) now parses `instanceId` from the `/api/instance/info` response (returning `{ reachable, instanceId }`; a missing/unparseable epoch is `null`, never an error). The shared decision helper `recoverOrDetectReset(peer, result)` — used by both the background recovery tick (`processRecoveryTick`) and the manual recheck endpoint — routes to `markPeerReset` (returning `'reset_detected'`) when the peer has a non-null `peer_instance_id` and the probed epoch differs, and **does NOT call `markPeerRecovered`**. Rationale: a genuinely reset peer's HMAC secret is desynced, so flipping it back to `active` via a reachability probe would resume relay against a dead secret. Only when the probed epoch matches the baseline (or the baseline is null / epoch unknown) does the normal recovery-to-active path run. The manual recheck endpoint returns `{ recovered: false, status: 'needs_attention' }` on `'reset_detected'`.
- **Health-tick probe (`needs_attention` peers)** — `detectResetOnNeedsAttentionPeers()` (`utils/federationRecovery.ts`), called from `processHealthCheckTick` on the 15-minute tick, closes design §4.1's remaining sub-case. A reset peer can reach `needs_attention` via the **auth-failure path** — its HTTP is up but returns 401/403 because the new incarnation has no peer row for us, so `consecutive_auth_failures` crosses `AUTH_FAILURE_THRESHOLD`**without ever transitioning through `unreachable`**. The `unreachable`-only recovery probe therefore never observes its epoch change, so no journal is ever created; a later manual Re-peer would then run `healResetIncarnation` with no journal row → no heal → the split-brain persists. This pass selects peers with `status='needs_attention'` AND `peer_instance_id IS NOT NULL` AND `needs_attention_reason` not already `peer_reset_detected` (those already carry a journal), probes each (`probePeerReachable`, one `/instance/info` GET per qualifying peer per tick), and calls `markPeerReset` on an observed epoch mismatch. **Detection only:** unlike `recoverOrDetectReset`, it NEVER flips a `needs_attention` peer to `active` (a match / unknown / unreachable result is a pure no-op) — that peer's secret is desynced and only an admin-authenticated re-peer restores trust. It touches neither `peer_instance_id` nor `hmac_secret`.
- **`needs_attention`-peer reset probe** — closes design §4.1's remaining sub-case. A reset peer can reach `needs_attention` via the **auth-failure path** — its HTTP is up but returns 401/403 because the new incarnation has no peer row for us, so `consecutive_auth_failures` crosses `AUTH_FAILURE_THRESHOLD`**without ever transitioning through `unreachable`**. The `unreachable`-only 5-second recovery probe therefore never observes its epoch change, so no journal is created; a later manual Re-peer would then run `healResetIncarnation` with no journal row → no heal → the split-brain persists. The shared per-peer unit is **`detectResetForPeer(peer)`** (`utils/federationRecovery.ts`): for a peer with a non-null baseline it runs one `probePeerReachable` (`/instance/info` GET) and calls `markPeerReset` on an observed epoch mismatch. It is invoked from **three** places, so detection latency is near-zero rather than up to a full health-check cycle:
1. **Event-driven, at the transition** — the instant the outbox worker moves a peer to `needs_attention` on the auth-failure threshold (`federationWorker.ts`), it fires `detectResetForPeer` for that peer (fire-and-forget). This is the common live case: the moment the connection is declared broken, the epoch is checked and "Re-peer & heal" surfaces immediately.
2. **Worker-startup sweep**`startFederationWorkers()` runs `detectResetOnNeedsAttentionPeers()` once on boot, catching any peer already parked in `needs_attention` (reset while this instance was down, or transitioned before this probe shipped).
3. **15-minute health-tick backstop**`detectResetOnNeedsAttentionPeers()` also still runs at the end of `processHealthCheckTick` as the periodic safety net. It selects peers with `status='needs_attention'` AND `peer_instance_id IS NOT NULL` AND `needs_attention_reason` not already `peer_reset_detected` (those already carry a journal) and calls `detectResetForPeer` on each.
**Detection only:** unlike `recoverOrDetectReset`, none of these ever flips a `needs_attention` peer to `active` (a match / unknown / unreachable result is a pure no-op) — that peer's secret is desynced and only an admin-authenticated re-peer restores trust. Neither `peer_instance_id` nor `hmac_secret` is touched.
Legacy peers advertise no epoch (`instanceId` null), so detection requires a non-null observed epoch differing from a non-null stored baseline — legacy peers never trigger it, and the existing `auth_failures → needs_attention → manual Reset` path continues unchanged for them.