chore: add system docs, specs, and misc updates from other sessions

- Add complete docs/systems/ reference (18 system docs) - Add federation relay status doc and prior spec/plan docs - Remove superseded docs/federation-dm-s2s.md (replaced by docs/systems/federation.md) - CLAUDE.md updates - Minor fixes in social.ts, types.ts, AddDmMemberModal, NewDmModal, UserSettings
2026-03-31 03:40:34 +02:00
parent 70678ff198
commit a3a7527c9e
25 changed files with 8414 additions and 1243 deletions
@@ -0,0 +1,460 @@
+# Embed & Link Preview System
+
+Source files:
+- `packages/server/src/utils/embedClassifier.ts` — URL classification and provider detection
+- `packages/server/src/utils/embedResolver.ts` — URL extraction, embed resolution pipeline, DB persistence, batch fetching
+- `packages/server/src/utils/metadataFetcher.ts` — OpenGraph/HTML metadata scraping with Cheerio
+- `packages/server/src/utils/ssrf.ts` — SSRF protection (DNS resolution, private IP blocking)
+- `packages/web/src/components/chat/EmbedRenderer.tsx` — Client-side embed routing by type
+- `packages/web/src/components/chat/embeds/GenericEmbed.tsx` — Generic link preview card
+- `packages/web/src/components/chat/embeds/ImageEmbed.tsx` — Direct image embed with lightbox
+- `packages/web/src/components/chat/embeds/RichEmbed.tsx` — Rich iframe embed (Spotify)
+- `packages/web/src/components/chat/embeds/VideoEmbed.tsx` — Video embed (YouTube, Vimeo, direct)
+- `packages/shared/src/types.ts` — `Embed`, `EmbedType`, `EmbedProvider` type definitions
+
+---
+
+## Type Definitions
+
+```typescript
+type EmbedType = 'generic' | 'video' | 'image' | 'audio' | 'rich';
+type EmbedProvider = 'youtube' | 'vimeo' | 'spotify';
+
+interface Embed {
+  id: string;               // Snowflake
+  messageId: string | null;   // FK -> messages.id (space messages)
+  dmMessageId: string | null; // FK -> dm_messages.id (DMs)
+  url: string;               // Original URL from message content
+  embedType: EmbedType;
+  provider: EmbedProvider | null;
+  title: string | null;
+  description: string | null;
+  image: string | null;      // Thumbnail / og:image URL
+  embedUrl: string | null;   // iframe-safe embed URL
+  width: number | null;      // Image/thumbnail pixel width
+  height: number | null;     // Image/thumbnail pixel height
+  color: string | null;      // Reserved, always null currently
+  createdAt: number;         // Epoch ms
+}
+```
+
+DB schema: see `embeds` table in [database.md](database.md). Constraint: exactly one of `messageId`/`dmMessageId` is set.
+
+---
+
+## Pipeline Overview
+
+```
+Message created/edited
+  -> extractUrls(content)         // regex, dedupe, limit 5
+  -> for each URL:
+       classifyUrl(url)           // extension match or provider detection
+       fetchUrlMetadata(url)      // if needsMetadataFetch (SSRF-validated)
+       probeRemoteImageDimensions // if image with unknown dimensions
+       INSERT into embeds table
+  -> broadcast embeds_resolved / dm_embeds_resolved via WebSocket
+```
+
+---
+
+## 1. URL Extraction
+
+`embedResolver.ts:extractUrls()`
+
+**Regex:** `https?:\/\/[^\s<>"{}|\\^`[\]]+`
+
+- Matches `http://` and `https://` URLs in message content
+- Deduplicates while preserving order (first occurrence wins)
+- **Limit:** 5 URLs per message (`MAX_EMBEDS_PER_MESSAGE = 5`)
+- Returns empty array for null/empty content
+
+---
+
+## 2. URL Classification
+
+`embedClassifier.ts:classifyUrl()`
+
+Classification runs in two phases: extension matching (no URL parsing needed), then provider matching (requires valid `URL` object).
+
+### Phase 1 — Direct Media Extensions
+
+Regex-based, checked before URL parsing. These skip metadata fetch entirely.
+
+| Pattern | EmbedType | Provider | needsMetadataFetch |
+|---------|-----------|----------|--------------------|
+| `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.avif` | `image` | null | false |
+| `.mp3`, `.ogg`, `.wav`, `.flac`, `.opus` | `audio` | null | false |
+| `.mp4`, `.webm`, `.mov` | `video` | null | false |
+
+Extension matching is case-insensitive and tolerates query strings (`(\?.*)?$`).
+
+### Phase 2 — Provider Matching
+
+Requires successful `new URL()` parsing. Hostname normalized by stripping `www.` prefix.
+
+#### YouTube
+
+Hosts: `youtube.com`, `m.youtube.com`, `youtu.be`
+
+Supported URL patterns via `extractYouTubeId()`:
+| Pattern | Example |
+|---------|---------|
+| `/watch?v=ID` | `youtube.com/watch?v=dQw4w9WgXcQ` |
+| `/shorts/ID` | `youtube.com/shorts/dQw4w9WgXcQ` |
+| `/embed/ID` | `youtube.com/embed/dQw4w9WgXcQ` |
+| `/v/ID` (legacy) | `youtube.com/v/dQw4w9WgXcQ` |
+| Short link | `youtu.be/dQw4w9WgXcQ` |
+
+Video ID regex: `[A-Za-z0-9_-]+`
+
+Result: `embedType: 'video'`, `provider: 'youtube'`, `embedUrl: https://www.youtube-nocookie.com/embed/{videoId}`, `needsMetadataFetch: true`
+
+Privacy: Uses `youtube-nocookie.com` domain for embed iframes.
+
+#### Vimeo
+
+Host: `vimeo.com`
+
+Pattern: `vimeo.com/{numericId}` (regex: `/^\/(\d+)/`)
+
+Result: `embedType: 'video'`, `provider: 'vimeo'`, `embedUrl: https://player.vimeo.com/video/{id}`, `needsMetadataFetch: true`
+
+#### Spotify
+
+Host: `open.spotify.com`
+
+Pattern: `open.spotify.com/{type}/{id}` where type is `track`, `album`, or `playlist`, id is `[A-Za-z0-9]+`
+
+Result: `embedType: 'rich'`, `provider: 'spotify'`, `embedUrl: https://open.spotify.com/embed/{type}/{id}`, `needsMetadataFetch: true`
+
+#### Fallthrough
+
+Any URL that does not match a provider: `embedType: 'generic'`, `provider: null`, `embedUrl: null`, `needsMetadataFetch: true`
+
+Invalid URLs (fail `new URL()` parsing): same as fallthrough.
+
+---
+
+## 3. SSRF Protection
+
+`ssrf.ts:validateExternalUrl()`
+
+Called before every outbound fetch (metadata fetching and image dimension probing). Throws on any violation.
+
+### Validation Steps
+
+1. **URL parsing** — `new URL(url)` must succeed
+2. **Scheme check** — only `http:` and `https:` allowed
+3. **DNS resolution** — `dns.promises.lookup(hostname)` resolves hostname to IP
+4. **Private IP check** — `isPrivateIp(address)` rejects internal addresses
+
+### Blocked IP Ranges
+
+`ssrf.ts:isPrivateIp()`
+
+| Range | Description |
+|-------|-------------|
+| `127.*` | Loopback |
+| `0.*`, `0.0.0.0` | Unspecified |
+| `10.*` | Private class A |
+| `192.168.*` | Private class C |
+| `172.16.0.0/12` | Private class B (172.16–172.31, checked via integer parse of second octet) |
+| `169.254.*` | Link-local |
+| `::1` | IPv6 loopback |
+| `fc*`, `fd*` | IPv6 unique local |
+| `fe80*` | IPv6 link-local |
+
+### Redirect Handling
+
+Both `fetchUrlMetadata` and `probeRemoteImageDimensions` use `redirect: 'follow'` in their `fetch()` calls. SSRF validation is performed on the **original** URL before fetch, but the native `fetch` follows redirects without re-validating intermediate URLs. This means a redirect from a public IP to a private IP would not be caught by the current implementation.
+
+---
+
+## 4. Metadata Fetching
+
+`metadataFetcher.ts:fetchUrlMetadata()`
+
+### Flow
+
+1. `validateExternalUrl(url)` — SSRF check, returns `null` on failure
+2. `fetch(url)` with `User-Agent: BackspaceBot/1.0`, 5-second timeout via `AbortController`
+3. **Content-Type detection** — if response is `image/*`, `video/*`, or `audio/*`, returns early with `contentType` field set (no HTML parsing)
+4. **Size guard** — rejects responses with `Content-Length > 512KB`
+5. **Stream-read with hard limit** — reads body via `ReadableStream`, stops at 512KB even for chunked (unknown-length) responses
+6. **HTML parsing** via Cheerio
+
+### Metadata Extraction
+
+Parsed from HTML using Cheerio with the following priority:
+
+| Field | Primary Source | Fallback |
+|-------|---------------|----------|
+| `title` | `og:title` | `<title>` element |
+| `description` | `og:description` | `<meta name="description">` |
+| `image` | `og:image` | none |
+| `siteName` | `og:site_name` | none |
+| `imageWidth` | `og:image:width` | none |
+| `imageHeight` | `og:image:height` | none |
+
+`imageWidth`/`imageHeight` are only included when they parse as finite positive integers.
+
+### Return Type
+
+```typescript
+interface UrlMetadata {
+  title: string | null;
+  description: string | null;
+  image: string | null;
+  siteName: string | null;
+  url: string;
+  contentType?: string;     // Set only for direct media (image/video/audio)
+  imageWidth?: number;      // From og:image:width (HTML pages only)
+  imageHeight?: number;     // From og:image:height (HTML pages only)
+}
+```
+
+---
+
+## 5. Embed Resolution
+
+`embedResolver.ts:resolveEmbeds()`
+
+Main pipeline function. Iterates over extracted URLs, resolves each independently (one URL failure does not block others).
+
+### Per-URL Resolution Logic
+
+```
+classify URL
+  |
+  |-- image (by extension)?
+  |     -> set image = url, skip metadata fetch
+  |
+  |-- needsMetadataFetch?
+  |     -> fetchUrlMetadata(url)
+  |     -> if metadata.contentType is media -> override embedType, set image = url
+  |     -> else -> extract title/description/image/dimensions from OG metadata
+  |     -> if generic + no title -> skip URL (no embed created)
+  |
+  |-- YouTube provider?
+  |     -> set fallback thumbnail: https://img.youtube.com/vi/{id}/hqdefault.jpg
+  |     -> set fallback dimensions: 480x360
+  |
+  |-- image with unknown dimensions?
+  |     -> probeRemoteImageDimensions(image)
+  |
+  -> INSERT embed row into DB
+```
+
+### Content-Type Override
+
+When `fetchUrlMetadata` returns a `contentType` field (indicating the URL points directly to a media file rather than an HTML page), the classifier's `embedType` is overridden:
+
+| Content-Type prefix | Override to |
+|---------------------|-------------|
+| `image/` | `image` (+ sets `image = url`) |
+| `video/` | `video` |
+| `audio/` | `audio` |
+
+When Content-Type is detected as media, OG metadata fields (title, description, image) are ignored.
+
+### YouTube Thumbnail Fallback
+
+For YouTube URLs, before metadata fetch, a predictable thumbnail URL is pre-populated:
+- URL: `https://img.youtube.com/vi/{videoId}/hqdefault.jpg`
+- Dimensions: 480x360 (hardcoded, matches hqdefault.jpg)
+
+If the metadata fetch returns an `og:image`, it does **not** override this fallback for `image` (the `||` operator means the pre-populated non-null value wins). However, OG dimensions would only populate `width`/`height` if they were still null (using `??`), so the hardcoded 480x360 persists.
+
+### Generic Embed Skip Rule
+
+If `effectiveEmbedType` remains `generic` after metadata fetch and no `title` was extracted, the URL is silently skipped — no embed row is created.
+
+### DB Insertion
+
+Each embed gets a unique Snowflake ID. The `messageId` / `dmMessageId` field is set based on the `isDm` flag. All inserts are synchronous (Drizzle `.run()`).
+
+---
+
+## 6. Image Dimension Probing
+
+`embedResolver.ts:probeRemoteImageDimensions()`
+
+Used for direct image URLs where dimensions are unknown (not provided by OG tags or provider defaults).
+
+### Mechanism
+
+1. **SSRF validation** — `validateExternalUrl(url)`, returns `null` on block
+2. **Range request** — fetches first 32KB (`PROBE_BYTES = 32_768`) with header `Range: bytes=0-32767`
+3. **Timeout** — 3-second abort (`PROBE_TIMEOUT_MS = 3_000`)
+4. **Graceful body read** — reads up to `PROBE_BYTES` via `ReadableStream`, then aggressively cancels the connection via `reader.cancel()`
+5. **Dimension extraction** — passes buffer to `sharp(buffer).metadata()`, returns `{width, height}` if both are positive integers
+
+### Headers
+
+```
+User-Agent: BackspaceBot/1.0
+Accept: image/*
+Range: bytes=0-32767
+```
+
+### Response Handling
+
+- Accepts HTTP `200` (server ignored Range) or `206` (partial content)
+- Any other status returns `null`
+- If the server returns more than 32KB (ignored Range header), only the first 32KB is read
+- All failures (network, timeout, unrecognized format, SSRF) silently return `null`
+
+---
+
+## 7. WebSocket Events
+
+### Broadcast Delivery
+
+After all embeds for a message are resolved, a single event is broadcast:
+
+| Context | Event Type | Delivery | Fields |
+|---------|-----------|----------|--------|
+| Space channel | `embeds_resolved` | `connectionManager.sendToChannel(spaceId, channelId, ...)` | `messageId`, `channelId`, `embeds[]` |
+| DM | `dm_embeds_resolved` | `connectionManager.sendToDmMembers(channelId, ...)` | `messageId`, `dmChannelId`, `embeds[]` |
+
+If no embeds were resolved (all URLs skipped or failed), no event is broadcast.
+
+### Client-Side Handling
+
+`useWebSocket.ts` handles both events by patching the message in `useChatStore`:
+1. Finds the message array for the channel/DM
+2. Maps over messages, replacing the `embeds` array for the matching `messageId`
+3. For federated contexts (`!isHome`): resolves relative image URLs via `resolveAssetUrl(embed.image, origin)`
+
+---
+
+## 8. Edit Re-resolution
+
+When a message is edited, embeds are re-resolved via a **delete-then-resolve** pattern (not using `reResolveEmbeds` — that function exists but is currently unused).
+
+### Edit Flow (identical for REST and WebSocket handlers)
+
+1. Update message content in DB
+2. **Synchronous delete** — `DELETE FROM embeds WHERE messageId = ?` (or `dmMessageId`)
+3. Broadcast `message_updated` / `dm_message_updated` with empty `embeds[]`
+4. **Asynchronous re-resolve** — `setImmediate(() => resolveEmbeds(...).catch(() => {}))` runs the full pipeline
+5. New embeds arrive via `embeds_resolved` / `dm_embeds_resolved` event
+
+This two-phase approach ensures the edit broadcast is immediate (with stale embeds removed), while new embeds arrive shortly after via a separate event.
+
+### Callsites
+
+| Handler | File | Line |
+|---------|------|------|
+| REST `PATCH /api/messages/:id` | `routes/messages.ts` | Inline delete + `setImmediate(resolveEmbeds)` |
+| WS `message_edit` | `ws/events.ts` | Inline delete + `setImmediate(resolveEmbeds)` |
+| REST `PATCH /api/dm/messages/:id` | `routes/dm.ts` | Inline delete + `setImmediate(resolveEmbeds)` |
+| WS `dm_message_edit` | `ws/events.ts` | Inline delete + `setImmediate(resolveEmbeds)` |
+
+---
+
+## 9. Batch Fetching
+
+Two functions load embeds for message lists (used when fetching message history):
+
+| Function | Filters by | Returns |
+|----------|-----------|---------|
+| `fetchEmbedsForMessages(messageIds[])` | `embeds.messageId IN (...)` | `Map<messageId, embedRow[]>` |
+| `fetchDmEmbedsForMessages(dmMessageIds[])` | `embeds.dmMessageId IN (...)` | `Map<dmMessageId, embedRow[]>` |
+
+`embedRowToEmbed()` converts a DB row to the shared `Embed` type (maps nulls, casts enum strings).
+
+---
+
+## 10. Client Rendering
+
+`EmbedRenderer` dispatches by `embed.embedType`:
+
+```
+EmbedRenderer
+  |-- 'video'   -> VideoEmbed
+  |-- 'image'   -> ImageEmbed
+  |-- 'audio'   -> inline <audio> element
+  |-- 'rich'    -> RichEmbed
+  |-- 'generic' -> GenericEmbed (default)
+```
+
+### VideoEmbed
+
+Two modes based on whether a provider is present:
+
+**Direct video** (`!provider && !embedUrl`):
+- Renders `<video>` element with `controls`, `preload="none"`, 16:9 aspect ratio
+
+**Provider iframe** (YouTube/Vimeo):
+- Initial state: thumbnail image with play button overlay (glass-bubble style)
+- On click: replaces with `<iframe>` loading `embedUrl?autoplay=1&origin={window.location.origin}`
+- iframe permissions: `accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share`, `allowFullScreen`
+- Footer shows capitalized provider name and linked title
+
+### ImageEmbed
+
+- Renders `<img>` with `max-w-[400px]`, `max-h-[300px]`, `loading="lazy"`, `referrerPolicy="no-referrer"`
+- Image source: `embed.image ?? embed.url`
+- Click opens image preview lightbox via `useUIStore.openImagePreview()`
+
+### RichEmbed (Spotify)
+
+Click-to-load pattern (no auto-loading of third-party iframes):
+
+**Unloaded state** (default):
+- Shows thumbnail, provider label, title, description, "Click to load" prompt
+- Entire card is a `<button>` that triggers load
+
+**Loaded state**:
+- Renders `<iframe>` with `sandbox="allow-scripts allow-same-origin allow-popups"`
+- iframe permissions: `autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture`
+
+Height determination via `getIframeHeight()`:
+1. `embed.height` if set
+2. `PROVIDER_HEIGHTS[provider]` — currently only `spotify: 152`
+3. Default: `200`
+
+### GenericEmbed
+
+- Returns `null` if no `embed.title` (no-op render)
+- Shows provider name (from `embed.provider` or parsed hostname), linked title, description (3-line clamp)
+- If `embed.image` exists: 80x80 thumbnail on the right side
+
+### AudioEmbed (inline in EmbedRenderer)
+
+- Shows linked title (if present) + native `<audio>` element with `controls`, `preload="metadata"`
+- Max width 400px, left border accent
+
+### Common Styling
+
+All embed cards share:
+- `max-w-[400px]` constraint
+- `mt-2` top margin (spacing from message content)
+- `bg-surface-channel` background (matte surface tier)
+- `rounded-[4px]` or `rounded-lg` corners
+- `referrerPolicy="no-referrer"` on all images
+
+---
+
+## 11. Utility Endpoint
+
+`GET /api/utils/metadata?url=` (auth required)
+
+Exposes `fetchUrlMetadata()` directly as a REST endpoint. Returns the `UrlMetadata` object, or `{}` if fetch fails. This is independent of the embed pipeline and can be used for ad-hoc URL previews.
+
+---
+
+## Constants Summary
+
+| Constant | Value | Location |
+|----------|-------|----------|
+| `MAX_EMBEDS_PER_MESSAGE` | 5 | `embedResolver.ts` |
+| `PROBE_BYTES` | 32,768 (32KB) | `embedResolver.ts` |
+| `PROBE_TIMEOUT_MS` | 3,000ms | `embedResolver.ts` |
+| Metadata fetch timeout | 5,000ms | `metadataFetcher.ts` |
+| HTML body size limit | 512,000 bytes (512KB) | `metadataFetcher.ts` |
+| User-Agent | `BackspaceBot/1.0` | both fetchers |
+| Spotify iframe height | 152px | `RichEmbed.tsx` |
+| Default rich iframe height | 200px | `RichEmbed.tsx` |
+| YouTube thumbnail dimensions | 480x360 | `embedResolver.ts` |