Merge branch 'feat/outbox-auth-failure-recovery'

Replace the federation outbox worker's 401/403 wipe-and-rehandshake loop with bounded retry (AUTH_FAILURE_THRESHOLD=5, ~21.5 min backoff window) and a new `needs_attention` peer state. Surfaces persistent HMAC desync to admins via a first-class 'Reset peering' action instead of the prior silent loop. Closes backlog item #19. Security invariants verified on live infra (Pi+VM): - hmac_secret is NEVER wiped in response to a network-observed 401/403 (Task 5 removes the wipe; Task 7 extends the /peer/accept idempotent- 200-no-update safeguard to cover needs_attention peers). - Auth failures increment only consecutive_auth_failures, never the network counter consecutive_failures (Task 5 splits handleOutboxDeliveryFailure → applyOutboxEntryBackoff). - Transition occurs at exactly 5 consecutive 401/403 responses; below threshold, entries get backoff but state is preserved; above, peer flips to needs_attention, affected users get federation_peer_rejected WS with 'Federation trust broken — admin must reset peering'. - /peer/accept safeguard confirmed against attacker curl probe on the live Pi instance while in needs_attention — forged-secret request returned 200-no-update, local hmac_secret unchanged. - Legitimate rotation (Scenario B) does not false-positive: both sides capture pending secret, auth_failures stays at 0, DM delivers cleanly during grace period. Four follow-up backlog items discovered during the work: #20 health-check cadence tightening (15-min grace vs 1-hour tick) #21 /peer/initiate 202 handling #22 consecutive_failures nullability normalization #23 unify client-side FederationPeer with shared type Spec: internal notes Plan: internal notes
2026-04-21 22:04:31 +02:00
parent 85668d4da8 e236d0730e
commit 9400189a8d
15 changed files with 3761 additions and 88 deletions
@@ -9,6 +9,7 @@ Source files:
 - `packages/server/src/utils/storageJanitor.ts` -- Storage stats, orphan detection, cleanup
 - `packages/web/src/stores/settingsStore.ts` -- Zustand store for instance/streaming settings
 - `packages/web/src/components/modals/instanceSettingsPanels/GeneralPanel.tsx` -- General settings UI
+- `packages/web/src/components/modals/instanceSettingsPanels/FederationPanel.tsx` -- Federation peers panel (peering, approval queue, peer status, rotation, reset)
 - `packages/web/src/components/modals/instanceSettingsPanels/StoragePanel.tsx` -- Storage management UI
 - `packages/web/src/components/modals/instanceSettingsPanels/StreamingPanel.tsx` -- Streaming config UI
 - `packages/web/src/components/modals/instanceSettingsPanels/UsersPanel.tsx` -- User management UI
@@ -422,14 +423,21 @@ All panels live under `packages/web/src/components/modals/instanceSettingsPanels

 #### GeneralPanel

-Manages: instance name, registration toggle, discovery toggle, GIF API key, federation relay toggle/TTL, pending approval requests, peered instances list.
+Manages: instance name, registration toggle, discovery toggle, GIF API key, federation relay toggle/TTL.

 - Instance name input: max 32 chars, enforced client-side via `slice(0, 32)`
 - GIF key: password input, separate dirty tracking (`gifKeyDirty`). Only sent on save if modified. "Clear key" button sets empty string.
+- Federation relay toggle and TTL input: drive `federationRelayEnabled` and `federationRelayTtlDays` instance settings.
+
+#### FederationPanel
+
+Manages: federation peers list, pending approval requests, manual peering initiation, secret rotation, peer reset.
+
 - **Pending Approvals section:** Visible only when `pendingApprovalCount > 0` (from ready payload). Positioned above the peer list. Each row shows the requesting instance name and origin with Approve and Deny buttons. Approve calls `api.federation.approveApprovalRequest(id)` and Deny calls `api.federation.denyApprovalRequest(id)`; both remove the row from the local list on success.
- Federation peers: fetched via `api.federation.peers()`, displayed as a list with status badges (active/pending/unreachable/awaiting_approval), last-seen/synced times, revoke button
- Peers with status `'revoked'` are filtered out of the visible list
- Revoke calls `api.federation.revokePeer(peerId)` and removes from local list
+- Federation peers: fetched via `api.federation.peers()`, displayed as a list with status badges (active/pending/unreachable/awaiting_approval/rejected/needs_attention), last-seen/synced times, and per-peer actions.
+- Peers with status `'revoked'` are filtered out of the visible list.
+- Revoke calls `api.federation.revokePeer(peerId)` and removes from local list.
+- Peers in `needs_attention` status render with a rose "Needs Attention" pill and a single "Reset Peering" action. The action opens a danger-variant ConfirmDialog explaining that reset deletes the local peer record (cascade-removes outbox entries) and requires out-of-band re-peering with the remote admin.

 #### StoragePanel

@@ -357,10 +357,11 @@ Migration flags (internal): `voice_bit_migrated`, `profile_attachments_cleaned`,
 | origin | text NOT NULL UNIQUE | | `https://domain.tld` |
 | instanceName | text | | |
 | hmacSecret | text NOT NULL | | 256-bit hex |
-| status | text NOT NULL | `'active'` | active/pending/awaiting_approval/unreachable/revoked/rejected |
+| status | text NOT NULL | `'active'` | active/pending/awaiting_approval/unreachable/revoked/rejected/needs_attention |
 | lastSeenAt | integer | | |
 | lastFailureAt | integer | | |
-| consecutiveFailures | integer | 0 | >=10 → unreachable |
+| consecutiveFailures | integer | 0 | >=10 → unreachable (network/5xx failures) |
+| consecutiveAuthFailures | integer NOT NULL | 0 | >=5 → needs_attention. Tracked separately from `consecutiveFailures` (network) because auth (401/403) and network failures have different resolution paths. |
 | lastSyncedAt | integer | 0 | |
 | remoteMaxUploadSize | integer | | Bytes, from peer |
 | createdAt | integer NOT NULL | | |
@@ -73,22 +73,26 @@ Both instances store the **same** HMAC secret. The initiating instance generates

 ```
                     ensurePeered
-  (none) ──────────► pending ──────────► active
-      ▲                │                   │
-      │                │  remote 202       │ delivery failures
-      │                ▼                   ▼
-      │          awaiting_approval     unreachable
-      │                │                   │
-      │    ┌───────────┼──────────┐        │ health check OK
-      │    │           │          │        ▼
-      │  accept     denied     expired   active
-      │  (fresh)    (admin)   (janitor)    │
-      │    │           │          │        │ admin revoke
-      │    ▼           ▼          ▼        ▼
-      │  active    rejected   rejected  revoked
-      │
-      │  auto-peer rejected (403 PEERING_REQUIRES_APPROVAL)
+  (none) ──────────► pending ──────────► active ──────► needs_attention
+      ▲                │                   │ ▲              │
+      │                │  remote 202       │ │              │ admin Reset
+      │                ▼                   │ │              ▼
+      │          awaiting_approval         │ │           (deleted)
+      │                │                   │ │
+      │    ┌───────────┼──────────┐        │ │ N consecutive
+      │    │           │          │        │ │ auth failures (401/403)
+      │  accept     denied     expired     │ │
+      │  (fresh)    (admin)   (janitor)    │ │ delivery failures
+      │    │           │          │        ▼ │
+      │    ▼           ▼          ▼     unreachable
+      │  active    rejected   rejected     │
+      │                                    │ health check OK
+      │  auto-peer rejected                ▼
+      │  (403 PEERING_REQUIRES_APPROVAL) active
      └──────────────────────────── rejected
+                                      │ admin revoke (active)
+                                      ▼
+                                   revoked
 ```

 | Status | Outbox delivery | Health check | Relay accepts | Re-initiation | Admin clear |
@@ -97,6 +101,7 @@ Both instances store the **same** HMAC secret. The initiating instance generates
 | `pending` | No | No | No | No (returns 409) | N/A |
 | `awaiting_approval` | No | No | No | Returns pending; no re-handshake | Yes (admin deletes) |
 | `unreachable` | No (entries wait) | Yes (1h interval) | Yes (resets to active) | No | N/A |
+| `needs_attention` | No (entries bounded by TTL) | No | Yes (200 no-update, same as active) | No (admin must Reset first) | Yes (admin Reset deletes record) |
 | `revoked` | No (entries purged) | No | No (returns 403) | Yes (old record deleted) | N/A |
 | `rejected` | No | No | No | Yes (admin deletes record, then re-initiates) | N/A |

@@ -151,6 +156,7 @@ When `autoAcceptPeering` is `false` and an instance calls `POST /api/federation/
 | `/api/federation/peers/:id` | PATCH | JWT + admin | Update peer settings (auto-rotation interval) |
 | `/api/federation/peers/:id` | DELETE | JWT + admin | Revoke peer, purge outbox |
 | `/api/federation/peers/:id/permanent` | DELETE | JWT + admin | Hard-delete revoked peer record |
+| `/api/federation/peers/:id/reset` | POST | JWT + admin | Delete peer record (cascade-deletes outbox). Only admissible in `needs_attention` state. |
 | `/api/federation/peers/:id/rotate` | POST | JWT + admin | Trigger immediate secret rotation |
 | `/api/federation/approval-requests` | GET | JWT + admin | List pending peering approval requests |
 | `/api/federation/approval-requests/:id/approve` | POST | JWT + admin | Approve request, initiate handshake |
@@ -480,6 +486,19 @@ Trigger (API/WS handler)
 | 6 | 6 hours |
 | 7+ | 24 hours (cap) |

+### Authentication-failure handling (401 / 403)
+
+When a relay response is 401 (HMAC rejected) or 403 (remote's peer row is non-active or missing), the worker increments `consecutive_auth_failures` on the peer row, applies backoff to the queued outbox entries via the existing `BACKOFF_SCHEDULE_MS`, and preserves `hmac_secret`. After `AUTH_FAILURE_THRESHOLD = 5` consecutive auth failures (~21.5 min with the existing schedule), the peer transitions to `needs_attention`:
+
+- Outbox delivery halts (the existing `status = 'active'` filter on the delivery query excludes `needs_attention`).
+- `hmac_secret` is preserved (admin can inspect; no silent rotation).
+- Affected local users receive `federation_peer_rejected` WS events with reason "Federation trust broken — admin must reset peering".
+- Admins receive `federation_peers_changed`.
+
+The worker NEVER re-handshakes via unauthenticated `/peer/accept` in response to a 401/403. The safeguard at `/peer/accept` (idempotent-200-no-update on `active` OR `needs_attention` peers) is what prevents silent HMAC rotation; the worker's job is to respect that signal and surface it to admins rather than loop. Recovery is via the admin "Reset peering" action, which deletes the local peer row and requires out-of-band re-peering.
+
+Network failures (timeouts, non-401/403 non-2xx responses) are tracked separately via `consecutive_failures` and lead to `unreachable` at `PEER_UNREACHABLE_THRESHOLD = 10`. A successful delivery resets both counters.
+
 ### Relay Request/Response Format

 **Request:**
@@ -0,0 +1 @@
+ALTER TABLE `federation_peers` ADD `consecutive_auth_failures` integer DEFAULT 0 NOT NULL;
@@ -22,6 +22,13 @@
      "when": 1776689384005,
      "tag": "0002_peer_approval_requests",
      "breakpoints": true
+    },
+    {
+      "idx": 3,
+      "version": "6",
+      "when": 1776789567488,
+      "tag": "0003_classy_loki",
+      "breakpoints": true
    }
  ]
 }
@@ -363,6 +363,7 @@ export const federationPeers = sqliteTable('federation_peers', {
  lastSeenAt: integer('last_seen_at'),
  lastFailureAt: integer('last_failure_at'),
  consecutiveFailures: integer('consecutive_failures').default(0),
+  consecutiveAuthFailures: integer('consecutive_auth_failures').notNull().default(0),
  lastSyncedAt: integer('last_synced_at').default(0),
  remoteMaxUploadSize: integer('remote_max_upload_size'),
  nonceSupported: integer('nonce_supported').notNull().default(0),
@@ -0,0 +1,193 @@
+import { describe, it, expect, beforeEach, vi } from 'vitest';
+import Fastify, { type FastifyInstance } from 'fastify';
+import Database from 'better-sqlite3';
+import { drizzle } from 'drizzle-orm/better-sqlite3';
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import * as schema from '../db/schema.js';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+type TestDb = ReturnType<typeof drizzle<typeof schema>>;
+let sqlite: Database.Database;
+let testDb: TestDb;
+let currentUserId = 'admin-user';
+let currentUserIsAdmin = true;
+
+// Mock getDb BEFORE importing the route module so the module reads our test DB.
+// Must also re-export `schema` because federation.ts imports it from '../db/index.js'.
+vi.mock('../db/index.js', () => ({
+  getDb: () => testDb,
+  schema,
+}));
+
+// Mock the auth middleware to honour test-controlled currentUserId / currentUserIsAdmin.
+vi.mock('../utils/auth.js', () => ({
+  authenticate: async (req: { userId?: string }) => {
+    if (!currentUserId) {
+      throw Object.assign(new Error('unauthenticated'), { statusCode: 401 });
+    }
+    req.userId = currentUserId;
+  },
+  requireAdmin: async (req: { userId?: string }, reply: { code: (n: number) => { send: (b: unknown) => unknown } }) => {
+    if (!currentUserIsAdmin) {
+      return reply.code(403).send({ error: 'Only instance admins can perform this action', statusCode: 403 });
+    }
+  },
+}));
+
+// Mock the WS connection manager — the handler only calls sendToAdmins.
+vi.mock('../ws/handler.js', () => ({
+  connectionManager: {
+    sendToAdmins: vi.fn(),
+    getAllOnlineUserIds: () => [],
+    sendToUser: vi.fn(),
+    sendToDmMembers: vi.fn(),
+  },
+}));
+
+async function buildApp(): Promise<FastifyInstance> {
+  const app = Fastify({ logger: false });
+  // Import the route module AFTER the mocks above are set up.
+  const { federationRoutes } = await import('./federation.js');
+  await app.register(federationRoutes);
+  await app.ready();
+  return app;
+}
+
+function applyMigrations(db: Database.Database): void {
+  const migrationsDir = path.resolve(__dirname, '../../drizzle');
+  const files = fs
+    .readdirSync(migrationsDir)
+    .filter(f => f.endsWith('.sql'))
+    .sort();
+  for (const f of files) {
+    const sql = fs.readFileSync(path.join(migrationsDir, f), 'utf8');
+    // drizzle-kit uses `--> statement-breakpoint` as separator
+    const statements = sql.split(/-->\s*statement-breakpoint/);
+    for (const stmt of statements) {
+      const clean = stmt.trim();
+      if (clean) db.exec(clean);
+    }
+  }
+}
+
+describe('POST /api/federation/peers/:id/reset', () => {
+  let app: FastifyInstance;
+
+  beforeEach(async () => {
+    sqlite = new Database(':memory:');
+    sqlite.pragma('foreign_keys = ON');
+    applyMigrations(sqlite);
+    testDb = drizzle(sqlite, { schema });
+    currentUserId = 'admin-user';
+    currentUserIsAdmin = true;
+    app = await buildApp();
+  });
+
+  it('returns 404 when the peer does not exist', async () => {
+    const res = await app.inject({
+      method: 'POST',
+      url: '/api/federation/peers/nonexistent/reset',
+    });
+    expect(res.statusCode).toBe(404);
+    expect(res.json()).toMatchObject({ error: 'Peer not found' });
+  });
+
+  it('returns 400 when the peer status is not needs_attention', async () => {
+    testDb.insert(schema.federationPeers).values({
+      id: 'peer-1',
+      origin: 'https://example.com',
+      hmacSecret: 'a'.repeat(64),
+      status: 'active',
+      createdAt: Date.now(),
+    }).run();
+
+    const res = await app.inject({
+      method: 'POST',
+      url: '/api/federation/peers/peer-1/reset',
+    });
+    expect(res.statusCode).toBe(400);
+    expect(res.json().error).toContain('needs_attention');
+
+    // Verify the peer row was NOT deleted
+    const still = testDb.select().from(schema.federationPeers).all();
+    expect(still).toHaveLength(1);
+  });
+
+  it('returns 403 when the caller is not an admin', async () => {
+    currentUserIsAdmin = false;
+    testDb.insert(schema.federationPeers).values({
+      id: 'peer-1',
+      origin: 'https://example.com',
+      hmacSecret: 'a'.repeat(64),
+      status: 'needs_attention',
+      createdAt: Date.now(),
+    }).run();
+
+    const res = await app.inject({
+      method: 'POST',
+      url: '/api/federation/peers/peer-1/reset',
+    });
+    expect(res.statusCode).toBe(403);
+
+    // Row must still exist
+    const still = testDb.select().from(schema.federationPeers).all();
+    expect(still).toHaveLength(1);
+  });
+
+  it('deletes the peer and cascade-removes outbox entries on success', async () => {
+    const now = Date.now();
+    testDb.insert(schema.federationPeers).values({
+      id: 'peer-1',
+      origin: 'https://example.com',
+      hmacSecret: 'a'.repeat(64),
+      status: 'needs_attention',
+      createdAt: now,
+    }).run();
+
+    // Queue two outbox entries for this peer
+    testDb.insert(schema.federationOutbox).values([
+      {
+        id: 'out-1',
+        peerId: 'peer-1',
+        contextId: 'dm-1',
+        entityId: 'msg-1',
+        contextType: 'dm',
+        eventType: 'message_create',
+        payload: '{}',
+        nextRetryAt: now,
+        expiresAt: now + 86_400_000,
+        createdAt: now,
+      },
+      {
+        id: 'out-2',
+        peerId: 'peer-1',
+        contextId: 'dm-1',
+        entityId: 'msg-2',
+        contextType: 'dm',
+        eventType: 'message_create',
+        payload: '{}',
+        nextRetryAt: now,
+        expiresAt: now + 86_400_000,
+        createdAt: now,
+      },
+    ]).run();
+
+    const res = await app.inject({
+      method: 'POST',
+      url: '/api/federation/peers/peer-1/reset',
+    });
+    expect(res.statusCode).toBe(200);
+    expect(res.json()).toEqual({ success: true });
+
+    // Peer deleted
+    const peers = testDb.select().from(schema.federationPeers).all();
+    expect(peers).toHaveLength(0);
+
+    // Outbox entries cascade-removed
+    const outbox = testDb.select().from(schema.federationOutbox).all();
+    expect(outbox).toHaveLength(0);
+  });
+});
@@ -491,8 +491,17 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
        .get();

      if (existing) {
-        if (existing.status === 'active') {
-          // Idempotent — already peered
+        if (existing.status === 'active' || existing.status === 'needs_attention') {
+          // Idempotent — already peered (or peering is in needs_attention state).
+          // In both cases we refuse to overwrite hmac_secret via this
+          // unauthenticated endpoint. An unauthenticated caller cannot
+          // prove prior trust, and needs_attention means "we don't know
+          // why this broke" — letting an unauthenticated request flip it
+          // to active with a new secret defeats the purpose.
+          //
+          // Legitimate recovery path: local admin clicks "Reset peering" →
+          // row is deleted → remote's /peer/accept then lands on a
+          // non-existent row and the normal handshake path runs.
          return reply.code(200).send({ accepted: true });
        }
        if (existing.status === 'revoked') {
@@ -836,6 +845,45 @@ export async function federationRoutes(app: FastifyInstance): Promise<void> {
    },
  );

+  // ─── POST /api/federation/peers/:id/reset ──────────────────────────────────
+  // Admin-only: reset a peer that has transitioned to needs_attention.
+  // Deletes the local peer row (cascade-deletes outbox entries via FK).
+  // Admin must re-initiate peering out of band after reset.
+  app.post<{ Params: { id: string } }>(
+    '/api/federation/peers/:id/reset',
+    { preHandler: [authenticate, requireAdmin] },
+    async (request, reply) => {
+      const { id } = request.params;
+      const db = getDb();
+
+      const peer = db
+        .select()
+        .from(schema.federationPeers)
+        .where(eq(schema.federationPeers.id, id))
+        .get();
+
+      if (!peer) {
+        return reply.code(404).send({ error: 'Peer not found', statusCode: 404 });
+      }
+
+      if (peer.status !== 'needs_attention') {
+        return reply.code(400).send({
+          error: 'Reset is only available for peers in the needs_attention state. Use revoke for active peers.',
+          statusCode: 400,
+        });
+      }
+
+      // Cascade-delete handles federation_outbox entries (FK onDelete: 'cascade').
+      db.delete(schema.federationPeers)
+        .where(eq(schema.federationPeers.id, id))
+        .run();
+
+      connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
+
+      return reply.code(200).send({ success: true });
+    },
+  );
+
  // ─── PATCH /api/federation/peers/:id ────────────────────────────────────────
  // Admin-only: update peer settings (e.g. auto-rotation interval).
  app.patch<{ Params: { id: string }; Body: { autoRotateIntervalDays?: number } }>(
@@ -0,0 +1,35 @@
+import { describe, it, expect } from 'vitest';
+import { evaluateAuthFailure, AUTH_FAILURE_THRESHOLD } from './federationAuthFailure.js';
+
+describe('evaluateAuthFailure', () => {
+  it('returns backoff with incremented count for a first failure', () => {
+    const result = evaluateAuthFailure(0);
+    expect(result).toEqual({ kind: 'backoff', newAuthFailures: 1 });
+  });
+
+  it('returns backoff below the threshold', () => {
+    for (let prev = 0; prev < AUTH_FAILURE_THRESHOLD - 1; prev++) {
+      const result = evaluateAuthFailure(prev);
+      expect(result.kind).toBe('backoff');
+      expect(result.newAuthFailures).toBe(prev + 1);
+    }
+  });
+
+  it('returns transition at the threshold', () => {
+    const result = evaluateAuthFailure(AUTH_FAILURE_THRESHOLD - 1);
+    expect(result).toEqual({
+      kind: 'transition_to_needs_attention',
+      newAuthFailures: AUTH_FAILURE_THRESHOLD,
+    });
+  });
+
+  it('returns transition beyond the threshold', () => {
+    const result = evaluateAuthFailure(AUTH_FAILURE_THRESHOLD + 5);
+    expect(result.kind).toBe('transition_to_needs_attention');
+    expect(result.newAuthFailures).toBe(AUTH_FAILURE_THRESHOLD + 6);
+  });
+
+  it('AUTH_FAILURE_THRESHOLD is 5', () => {
+    expect(AUTH_FAILURE_THRESHOLD).toBe(5);
+  });
+});
@@ -0,0 +1,28 @@
+/**
+ * Number of consecutive 401/403 responses from an active peer before the
+ * outbox worker transitions that peer to `needs_attention`.
+ *
+ * Rationale (see design spec §Retry Budget): with the existing
+ * BACKOFF_SCHEDULE_MS = [30s, 1m, 5m, 15m, 1h], five consecutive auth
+ * failures span ~21.5 min — covering the 15-min rotation grace window
+ * with ~6.5 min margin while still giving clear signal that a persistent
+ * desync has occurred.
+ */
+export const AUTH_FAILURE_THRESHOLD = 5;
+
+export type AuthFailureAction =
+  | { kind: 'backoff'; newAuthFailures: number }
+  | { kind: 'transition_to_needs_attention'; newAuthFailures: number };
+
+/**
+ * Pure decision function. Given the current consecutive-auth-failure count,
+ * return whether the next failure keeps the peer in retry-with-backoff or
+ * transitions it to the `needs_attention` terminal state.
+ */
+export function evaluateAuthFailure(currentAuthFailures: number): AuthFailureAction {
+  const newAuthFailures = currentAuthFailures + 1;
+  if (newAuthFailures >= AUTH_FAILURE_THRESHOLD) {
+    return { kind: 'transition_to_needs_attention', newAuthFailures };
+  }
+  return { kind: 'backoff', newAuthFailures };
+}
@@ -5,6 +5,7 @@ import { config } from '../config.js';
 import { isFederationRelayEnabled, queueOutboxEvent } from './federationOutbox.js';
 import { runFederationJanitor } from './storageJanitor.js';
 import { buildFederationHeaders, getOurOrigin, generateHmacSecret, ROTATION_GRACE_PERIOD_MS } from './federationAuth.js';
+import { evaluateAuthFailure, AUTH_FAILURE_THRESHOLD } from './federationAuthFailure.js';
 import { generateSnowflake } from './snowflake.js';
 import { getDmMessageWithUser } from '../routes/dm.js';
 import { connectionManager } from '../ws/handler.js';
@@ -261,29 +262,65 @@ async function processOutboxTick(): Promise<void> {
          .set({
            lastSeenAt: now,
            consecutiveFailures: 0,
+            consecutiveAuthFailures: 0,
          })
          .where(eq(schema.federationPeers.id, peerId))
          .run();
      } else if (response.status === 401 || response.status === 403) {
-        // 401 = HMAC verification failed (remote deleted our peer record entirely)
-        // 403 = Peer exists but is not active (remote revoked/rejected us)
-        // Both mean the peer relationship is broken on the remote side. Reset to
-        // 'pending' so resolvePendingPeers() triggers a fresh handshake via
-        // ensurePeered() on the next tick.
-        // Outbox entries are preserved (same peer ID) and will deliver after re-peering.
-        console.warn(
-          `[federation-worker] Peer ${peerOrigin} returned ${response.status} (peer stale) — resetting to pending for re-handshake`,
-        );
-        db.update(schema.federationPeers)
-          .set({
-            status: 'pending',
-            hmacSecret: '',  // Will be regenerated by ensurePeered/performHandshake
-            consecutiveFailures: 0,
-            lastFailureAt: now,
-          })
+        // HMAC rejected or remote's peer row non-active. Do NOT re-handshake
+        // via the unauthenticated /peer/accept path — the remote's
+        // idempotent-200-no-update safeguard would loop forever and, more
+        // importantly, re-handshaking in response to a 401 is not how trust
+        // gets healed. Persistent auth failures transition to
+        // needs_attention; bounded retry (AUTH_FAILURE_THRESHOLD) rides out
+        // transient clock skew and rotation-grace edge races.
+        const currentRow = db
+          .select({ consecutiveAuthFailures: schema.federationPeers.consecutiveAuthFailures })
+          .from(schema.federationPeers)
          .where(eq(schema.federationPeers.id, peerId))
-          .run();
-        connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
+          .get();
+        const decision = evaluateAuthFailure(currentRow?.consecutiveAuthFailures ?? 0);
+
+        if (decision.kind === 'transition_to_needs_attention') {
+          db.update(schema.federationPeers)
+            .set({
+              status: 'needs_attention',
+              consecutiveAuthFailures: decision.newAuthFailures,
+              lastFailureAt: now,
+            })
+            .where(eq(schema.federationPeers.id, peerId))
+            .run();
+          console.warn(
+            `[federation-worker] Peer ${peerOrigin} transitioned to needs_attention after ${decision.newAuthFailures} consecutive ${response.status} responses`,
+          );
+
+          const contextMap = buildContextMapForPeer(db, peerId);
+          if (contextMap.size > 0) {
+            pushPeerRejectedEvent(
+              peerOrigin,
+              contextMap,
+              'Federation trust broken — admin must reset peering',
+            );
+          }
+          connectionManager.sendToAdmins({ type: 'federation_peers_changed' as const });
+        } else {
+          // Below threshold — preserve state, apply backoff to outbox entries.
+          // Do NOT call handleOutboxDeliveryFailure here: per spec, auth failures
+          // must NOT increment consecutive_failures (that counter drives the
+          // 'unreachable' transition, which is a network-layer signal, not an
+          // auth-layer one).
+          console.warn(
+            `[federation-worker] Peer ${peerOrigin} returned ${response.status} (auth failure ${decision.newAuthFailures}/${AUTH_FAILURE_THRESHOLD})`,
+          );
+          db.update(schema.federationPeers)
+            .set({
+              consecutiveAuthFailures: decision.newAuthFailures,
+              lastFailureAt: now,
+            })
+            .where(eq(schema.federationPeers.id, peerId))
+            .run();
+          applyOutboxEntryBackoff(db, peerEntries, now);
+        }
      } else {
        console.warn(
          `[federation-worker] Peer ${peerOrigin} returned HTTP ${response.status}`,
@@ -307,13 +344,11 @@ async function processOutboxTick(): Promise<void> {
  await resolvePendingPeers();
 }

-function handleOutboxDeliveryFailure(
+function applyOutboxEntryBackoff(
  db: ReturnType<typeof getDb>,
-  peerId: string,
  entries: Array<{ outboxId: string; attempts: number | null }>,
  now: number,
 ): void {
-  // Increment attempts and compute next retry for each entry
  for (const entry of entries) {
    const newAttempts = (entry.attempts ?? 0) + 1;
    const backoffMs = getBackoffMs(newAttempts);
@@ -326,8 +361,18 @@ function handleOutboxDeliveryFailure(
      .where(eq(schema.federationOutbox.id, entry.outboxId))
      .run();
  }
+}

-  // Update peer failure tracking
+function handleOutboxDeliveryFailure(
+  db: ReturnType<typeof getDb>,
+  peerId: string,
+  entries: Array<{ outboxId: string; attempts: number | null }>,
+  now: number,
+): void {
+  applyOutboxEntryBackoff(db, entries, now);
+
+  // Update peer failure tracking (network/generic-error path only — auth failures
+  // use consecutive_auth_failures instead).
  const peer = db
    .select({ consecutiveFailures: schema.federationPeers.consecutiveFailures })
    .from(schema.federationPeers)
@@ -396,22 +441,7 @@ async function resolvePendingPeers(): Promise<void> {
      case 'rejected': {
        console.warn(`[federation-worker] Auto-peering rejected by ${peerOrigin}: ${result.error}`);

-        // Collect affected contexts before purging
-        const entries = db
-          .select({
-            contextId: schema.federationOutbox.contextId,
-            contextType: schema.federationOutbox.contextType,
-          })
-          .from(schema.federationOutbox)
-          .where(eq(schema.federationOutbox.peerId, peerId))
-          .all();
-
-        const contextMap = new Map<string, string>();
-        for (const e of entries) {
-          if (!contextMap.has(e.contextId)) {
-            contextMap.set(e.contextId, e.contextType);
-          }
-        }
+        const contextMap = buildContextMapForPeer(db, peerId);

        // Purge outbox entries (NOT mutation log)
        db.delete(schema.federationOutbox)
@@ -433,6 +463,34 @@ async function resolvePendingPeers(): Promise<void> {
  }
 }

+/**
+ * Build a map of contextId → contextType for all outbox entries targeting
+ * a specific peer. Used for surfacing "delivery impossible" via
+ * pushPeerRejectedEvent when a peer is rejected or transitioned to
+ * needs_attention.
+ */
+function buildContextMapForPeer(
+  db: ReturnType<typeof getDb>,
+  peerId: string,
+): Map<string, string> {
+  const entries = db
+    .select({
+      contextId: schema.federationOutbox.contextId,
+      contextType: schema.federationOutbox.contextType,
+    })
+    .from(schema.federationOutbox)
+    .where(eq(schema.federationOutbox.peerId, peerId))
+    .all();
+
+  const contextMap = new Map<string, string>();
+  for (const e of entries) {
+    if (!contextMap.has(e.contextId)) {
+      contextMap.set(e.contextId, e.contextType);
+    }
+  }
+  return contextMap;
+}
+
 /**
 * Push a federation_peer_rejected WS event to all local users affected by
 * the rejection. Resolves contextLabel from the database for each context.
@@ -986,10 +986,14 @@ export interface FederationPeer {
  id: string;
  origin: string;
  instanceName: string | null;
-  status: 'pending' | 'active' | 'unreachable' | 'revoked';
+  status: 'pending' | 'active' | 'unreachable' | 'revoked' | 'rejected' | 'awaiting_approval' | 'needs_attention';
  lastSeenAt: number | null;
  lastFailureAt: number | null;
  consecutiveFailures: number;
+  consecutiveAuthFailures: number;
  lastSyncedAt: number;
+  autoRotateIntervalDays: number;
+  secretRotatedAt: number | null;
+  rotationInProgress: boolean;
  createdAt: number;
 }
@@ -71,6 +71,7 @@ export interface FederationPeer {
  lastSeenAt: number | null;
  lastFailureAt: number | null;
  consecutiveFailures: number | null;
+  consecutiveAuthFailures: number;
  lastSyncedAt: number | null;
  createdAt: number;
  secretRotatedAt: number | null;
@@ -235,6 +236,7 @@ export class BackspaceApiClient {
    ensurePeered: (data: { remoteOrigin: string }) => Promise<{ peeringStatus: string; peerId?: string; error?: string }>;
    peers: () => Promise<{ peers: FederationPeer[] }>;
    revokePeer: (id: string) => Promise<{ success: boolean }>;
+    resetPeer: (id: string) => Promise<{ success: boolean }>;
    rotatePeerSecret: (id: string) => Promise<{ success: boolean; gracePeriodMs: number }>;
    updatePeer: (id: string, data: { autoRotateIntervalDays: number }) => Promise<{ peer: FederationPeer }>;
    deletePeerPermanently: (id: string) => Promise<{ success: boolean }>;
@@ -688,6 +690,8 @@ export class BackspaceApiClient {
        ),
      revokePeer: (id: string) =>
        request<{ success: boolean }>('DELETE', `/federation/peers/${id}`),
+      resetPeer: (id: string) =>
+        request<{ success: boolean }>('POST', `/federation/peers/${id}/reset`),
      rotatePeerSecret: (id: string) =>
        request<{ success: boolean; gracePeriodMs: number }>('POST', `/federation/peers/${id}/rotate`),
      updatePeer: (id: string, data: { autoRotateIntervalDays: number }) =>
@@ -175,6 +175,7 @@ function peerStatusColor(status: string): string {
    case 'unreachable': return 'bg-accent-amber/15 text-accent-amber';
    case 'rejected': return 'bg-accent-rose/15 text-accent-rose';
    case 'awaiting_approval': return 'bg-accent-amber/15 text-accent-amber';
+    case 'needs_attention': return 'bg-accent-rose/15 text-accent-rose';
    case 'revoked': return 'bg-white/5 text-txt-tertiary';
    default: return 'bg-white/5 text-txt-tertiary';
  }
@@ -187,6 +188,7 @@ function peerStatusDotColor(status: string): string {
    case 'unreachable': return 'bg-accent-amber';
    case 'rejected': return 'bg-accent-rose';
    case 'awaiting_approval': return 'bg-accent-amber';
+    case 'needs_attention': return 'bg-accent-rose';
    default: return 'bg-txt-tertiary';
  }
 }
@@ -199,13 +201,14 @@ function peerStatusLabel(status: string): string {
    case 'rejected': return 'Rejected (auto-peering denied)';
    case 'revoked': return 'Revoked';
    case 'awaiting_approval': return 'Awaiting Approval';
+    case 'needs_attention': return 'Needs Attention';
    default: return status;
  }
 }

 type PeerView = 'active' | 'revoked';
 type SortBy = 'name' | 'lastSeen' | 'dateAdded' | 'failures';
-type StatusFilter = 'active' | 'unreachable' | 'pending' | 'rejected' | 'awaiting_approval';
+type StatusFilter = 'active' | 'unreachable' | 'pending' | 'rejected' | 'awaiting_approval' | 'needs_attention';

 // ─── Filter Dropdown ─────────────────────────────────────────────────────────

@@ -267,7 +270,7 @@ function FilterDropdown({
            {view === 'active' && (
              <>
                <div className="text-[10px] font-semibold text-txt-tertiary uppercase tracking-wider px-2 py-1">Status</div>
-                {(['active', 'unreachable', 'pending', 'rejected', 'awaiting_approval'] as StatusFilter[]).map((s) => (
+                {(['active', 'unreachable', 'pending', 'rejected', 'awaiting_approval', 'needs_attention'] as StatusFilter[]).map((s) => (
                  <button
                    key={s}
                    type="button"
@@ -277,7 +280,11 @@ function FilterDropdown({
                    } hover:bg-white/[0.06] transition-colors`}
                  >
                    <div className={`w-2 h-2 rounded-full ${peerStatusDotColor(s)}`} />
-                    <span className="capitalize">{s === 'awaiting_approval' ? 'Awaiting Approval' : s}</span>
+                    <span className="capitalize">
+                      {s === 'awaiting_approval' ? 'Awaiting Approval'
+                        : s === 'needs_attention' ? 'Needs Attention'
+                        : s}
+                    </span>
                  </button>
                ))}
                <div className="h-px bg-white/[0.06] my-1" />
@@ -386,7 +393,7 @@ function PeerRow({ peer, view, expanded, onToggleExpand, onAction, defaultAutoRo
  view: PeerView;
  expanded: boolean;
  onToggleExpand: () => void;
-  onAction: (type: 'rotate' | 'revoke' | 'reinitiate' | 'delete') => void;
+  onAction: (type: 'rotate' | 'revoke' | 'reinitiate' | 'delete' | 'reset') => void;
  defaultAutoRotateIntervalDays: number;
 }) {
  const [editingInterval, setEditingInterval] = useState(false);
@@ -484,6 +491,14 @@ function PeerRow({ peer, view, expanded, onToggleExpand, onAction, defaultAutoRo
                      {peer.consecutiveFailures ?? 0}
                    </div>
                  </div>
+                  {peer.status === 'needs_attention' && (
+                    <div>
+                      <div className="text-[10px] text-txt-tertiary uppercase tracking-wider mb-0.5">Auth Failures</div>
+                      <div className="text-xs text-accent-rose font-medium">
+                        {peer.consecutiveAuthFailures}
+                      </div>
+                    </div>
+                  )}
                  <div>
                    <div className="text-[10px] text-txt-tertiary uppercase tracking-wider mb-0.5">Last Failure</div>
                    <div className="text-xs text-txt-secondary">{formatRelativeTime(peer.lastFailureAt)}</div>
@@ -547,30 +562,42 @@ function PeerRow({ peer, view, expanded, onToggleExpand, onAction, defaultAutoRo

                {/* Actions */}
                <div className="flex items-center gap-2">
-                  <button
-                    type="button"
-                    onClick={(e) => { e.stopPropagation(); onAction('rotate'); }}
-                    disabled={peer.rotationInProgress}
-                    className="px-3 py-1.5 text-xs font-medium bg-accent-lavender/10 text-accent-lavender hover:bg-accent-lavender/20 rounded transition-colors disabled:opacity-50"
-                    title={peer.rotationInProgress ? 'Rotation already in progress' : undefined}
-                  >
-                    Rotate Secret
-                  </button>
-                  <button
-                    type="button"
-                    onClick={(e) => { e.stopPropagation(); onAction('revoke'); }}
-                    className="px-3 py-1.5 text-xs font-medium bg-accent-rose/10 text-txt-danger hover:bg-accent-rose/20 rounded transition-colors"
-                  >
-                    Revoke
-                  </button>
-                  {!editingInterval && (
+                  {peer.status === 'needs_attention' ? (
                    <button
                      type="button"
-                      onClick={(e) => { e.stopPropagation(); setEditingInterval(true); setIntervalDraft(peer.autoRotateIntervalDays); }}
-                      className="text-[11px] text-txt-tertiary hover:text-txt-secondary underline decoration-dotted transition-colors ml-1"
+                      onClick={(e) => { e.stopPropagation(); onAction('reset'); }}
+                      className="px-3 py-1.5 text-xs font-medium bg-accent-rose/10 text-txt-danger hover:bg-accent-rose/20 rounded transition-colors"
                    >
-                      Edit rotation interval
+                      Reset Peering
                    </button>
+                  ) : (
+                    <>
+                      <button
+                        type="button"
+                        onClick={(e) => { e.stopPropagation(); onAction('rotate'); }}
+                        disabled={peer.rotationInProgress}
+                        className="px-3 py-1.5 text-xs font-medium bg-accent-lavender/10 text-accent-lavender hover:bg-accent-lavender/20 rounded transition-colors disabled:opacity-50"
+                        title={peer.rotationInProgress ? 'Rotation already in progress' : undefined}
+                      >
+                        Rotate Secret
+                      </button>
+                      <button
+                        type="button"
+                        onClick={(e) => { e.stopPropagation(); onAction('revoke'); }}
+                        className="px-3 py-1.5 text-xs font-medium bg-accent-rose/10 text-txt-danger hover:bg-accent-rose/20 rounded transition-colors"
+                      >
+                        Revoke
+                      </button>
+                      {!editingInterval && (
+                        <button
+                          type="button"
+                          onClick={(e) => { e.stopPropagation(); setEditingInterval(true); setIntervalDraft(peer.autoRotateIntervalDays); }}
+                          className="text-[11px] text-txt-tertiary hover:text-txt-secondary underline decoration-dotted transition-colors ml-1"
+                        >
+                          Edit rotation interval
+                        </button>
+                      )}
+                    </>
                  )}
                </div>
              </>
@@ -750,13 +777,15 @@ export function FederationPanel({ onApprovalCountChange }: { onApprovalCountChan
  const [peersLoading, setPeersLoading] = useState(false);
  const [peersError, setPeersError] = useState('');
  const [view, setView] = useState<PeerView>('active');
-  const [statusFilter, setStatusFilter] = useState<Set<StatusFilter>>(new Set(['active', 'unreachable', 'pending', 'rejected', 'awaiting_approval']));
+  const [statusFilter, setStatusFilter] = useState<Set<StatusFilter>>(
+    new Set(['active', 'unreachable', 'pending', 'rejected', 'awaiting_approval', 'needs_attention']),
+  );
  const [sortBy, setSortBy] = useState<SortBy>('name');
  const [expandedPeerId, setExpandedPeerId] = useState<string | null>(null);

  // Confirm dialog state (used in Task 10)
  const [confirmAction, setConfirmAction] = useState<{
-    type: 'rotate' | 'revoke' | 'reinitiate' | 'delete';
+    type: 'rotate' | 'revoke' | 'reinitiate' | 'delete' | 'reset';
    peer: FederationPeer;
  } | null>(null);
  const [actionLoading, setActionLoading] = useState(false);
@@ -855,6 +884,12 @@ export function FederationPanel({ onApprovalCountChange }: { onApprovalCountChan
          addToast('Peer permanently deleted', 'success', 2000);
          break;
        }
+        case 'reset': {
+          await api.federation.resetPeer(peer.id);
+          setPeers((prev) => prev.filter((p) => p.id !== peer.id));
+          addToast(`Peering reset for ${peer.instanceName || peer.origin}`, 'success', 3000);
+          break;
+        }
      }
    } catch (err) {
      addToast(err instanceof Error ? err.message : 'Action failed', 'warning', 3000);
@@ -891,6 +926,12 @@ export function FederationPanel({ onApprovalCountChange }: { onApprovalCountChan
        confirmLabel: 'Delete',
        variant: 'danger' as const,
      };
+      case 'reset': return {
+        title: 'Reset Peering',
+        description: `Reset peering with ${name}? This deletes the local peer record and all pending outbox entries. You must re-initiate peering with the remote admin out of band after reset. This cannot be undone.`,
+        confirmLabel: 'Reset',
+        variant: 'danger' as const,
+      };
    }
  })() : null;
				`@@ -0,0 +1 @@`
				ALTER TABLE `federation_peers` ADD `consecutive_auth_failures` integer DEFAULT 0 NOT NULL;