wip(mobile): in-progress mobile + voice polish
Snapshot of in-progress work parked here so group-DM-polish can land cleanly on main. Touches MainContent + mobile-ui.md which overlap with group-DM-polish; rebase onto post-merge main and resolve conflicts on those two files manually. Files: MainContent, MessageInput, MobileVoiceFullScreen (+test), StreamTile, VoiceUser, useLiveKit, useVisualViewportInset, AudioManager, voiceStore, voice utils, mobile-ui.md, voice.md, mobile-parity handoff doc.
This commit is contained in:
+40
-14
@@ -495,27 +495,52 @@ File: `MobileVoiceFullScreen.tsx`
|
||||
|
||||
**Header:** Collapse chevron (down arrow, pops screen), channel name, space name subtitle, participant count, members button (space channels only).
|
||||
|
||||
**Participant grid:**
|
||||
- `grid-cols-1` for 1-2 participants, `grid-cols-2` for 3+
|
||||
- Avatar size: 80px for 1-2 participants, 56px for 3+
|
||||
- Mute/deafen badge overlay on avatar (bottom-right, rose circle with icon)
|
||||
- Shows self-mute, space mute, permission mute, self-deafen, space deafen
|
||||
- Context menu on other participants: voice mod items, local mute checkbox, volume slider
|
||||
**Participant grid:** **Renders `<VoiceGrid participants={participants} />` from `packages/web/src/components/voice/VoiceGrid.tsx` — the same component desktop uses.** This is the source of the camera + screen-share rendering pipeline; mobile has no separate tile components. Reusing `VoiceGrid` gives mobile feature parity with desktop for free:
|
||||
|
||||
**Control bar:** `glass-bubble` container with safe area padding.
|
||||
- Camera tracks (`p.videoTrack` from `ParticipantInfo`) attach to a `<video>` element via LiveKit's `Track.attach(el)` so the SFU adaptive-stream observer can downshift simulcast layers based on the tile's painted pixel size — automatically scaling quality down on phone-shaped tiles.
|
||||
- Local camera: the local participant's `videoTrack` is attached identically, with `muted` on the `<video>` element so the user's own camera doesn't echo through their speakers. (`VoiceUser` sets `muted={isLocal}`.)
|
||||
- Remote screen-share tracks render in their own `StreamTile` (one extra tile per streaming participant) — the user must tap "Watch Stream" or focus the tile to subscribe; until then it's an avatar placeholder. `setStreamSubscription` and the `stream_watch` data-channel protocol fire identically on mobile.
|
||||
- Tap-to-focus: tapping any tile sets `voiceStore.focusedParticipantId` and the grid switches into the focused-publisher layout (one large tile + bottom strip of others). This works through touch events without modification.
|
||||
- Mute / deafen / camera badges, speaking-ring, "(you)" suffix, context-menu (right-click on desktop, long-press on iOS — Safari fires `contextmenu` on long-press), local-mute/volume sliders, watch/unwatch — all carry over.
|
||||
|
||||
| Button | State Colors |
|
||||
|--------|-------------|
|
||||
| Mute | Active: `bg-accent-rose/20 text-accent-rose`, Inactive: `bg-surface-elevated text-txt-primary` |
|
||||
| Deafen | Same as mute |
|
||||
| Camera | Active: `bg-accent-mint/20 text-accent-mint`, Inactive: same |
|
||||
| Screen share | Same as camera |
|
||||
| Disconnect | Always `bg-accent-rose text-white` |
|
||||
**Auto-focus on screen-share (mobile-only).** When `MobileVoiceFullScreen` mounts (or while it's already mounted) and a screen-share publication appears, the screen sets `focusedParticipantId` to the first live `${identity}:stream` tile so the user lands directly on the watchable stream. Two refs gate the behaviour:
|
||||
|
||||
- `userTouchedFocusRef` — flips `true` the first time `focusedParticipantId` changes to anything other than the auto-focused key, or to `null` after auto-focus had been set. Once flipped it stays flipped for the screen lifetime; auto-focus bails out. This means a user who explicitly dismisses focus via the Grid button never has it forced back on, even if a new screen-share starts.
|
||||
- `lastAutoFocusedKeyRef` — records the key we last auto-focused so the user-interaction detection can distinguish "user picked a different tile" from "we just set it ourselves".
|
||||
|
||||
The unmount cleanup clears `focusedParticipantId` so re-entering the call screen is a clean slate. Desktop is unaffected — auto-focus lives in `MobileVoiceFullScreen`, not `VoiceGrid`.
|
||||
|
||||
**Control bar:** `glass-bubble` container with safe area padding. Five round buttons.
|
||||
|
||||
| Button | Action | State Colors |
|
||||
|--------|--------|-------------|
|
||||
| Mute | `voiceStore.toggleMic` | Active: `bg-accent-rose/20 text-accent-rose`, Inactive: `bg-surface-elevated text-txt-primary` |
|
||||
| Deafen | `voiceStore.toggleDeafen` | Same as mute |
|
||||
| Camera | `handleCameraAction` (canonical, from `utils/voiceActions`) | Active: `bg-accent-mint/20 text-accent-mint`, Inactive: same |
|
||||
| Screen share | `handleScreenShareAction` (canonical, from `utils/voiceActions`) | Same as camera |
|
||||
| Disconnect | DM call or space voice teardown + pop screen | Always `bg-accent-rose text-white` |
|
||||
|
||||
**Screen-share button wiring (load-bearing).** The button calls `handleScreenShareAction`, **not** `voiceStore.toggleScreenShare`. The store action only flips the boolean — it never calls `getDisplayMedia` or publishes a track. `handleScreenShareAction` is the canonical path (also used by `VoiceControlBar` and the keybind handler) that calls `startScreenShare(room)` / `stopScreenShare(room)` and broadcasts voice status. iOS Safari does not support `getDisplayMedia` and the call rejects there — that's a platform limitation, not a Backspace bug; the same call behaves identically on desktop and Android.
|
||||
|
||||
**In-call camera switcher (mobile-only).** When `isCameraOn === true` AND `enumerateDevices()` returns more than one `videoinput`, a small chevron pill ("Switch camera", `aria-haspopup="menu"`) is overlaid on the top-right corner of the camera control button. Tapping it opens an upward-expanding menu portaled to `document.body` (so the popup escapes the control bar's `glass-bubble` clipping), pinned to the chevron's screen rect via `getBoundingClientRect()` and re-pinned on resize / capturing scroll. The menu lists every videoinput plus an "Auto (system default)" entry; the current selection is highlighted via `aria-checked` + a `text-txt-primary` accent. Selecting an entry calls `voiceStore.setCameraDeviceId(deviceId)` and closes the picker. The `useLiveKit` `syncCamera` effect picks up the store change and calls `room.switchActiveDevice('videoinput', target)` for an in-place hot-swap (no republish) — see `docs/systems/voice.md` "Hot-swap mid-call". Click-outside dismissal listens to both `mousedown` AND `touchstart` (iOS Safari does not synthesize `mousedown` reliably from a single tap).
|
||||
|
||||
The chevron is gated on `isCameraOn && cameraDevices.length > 1` so single-camera phones / desktops never see it. On iOS Safari the videoinput list is populated only after the OS-level camera permission has been granted at least once in the current session — that grant happens when the user first turns on the camera via `handleCameraAction` (which calls `getUserMedia`), so by the time the chevron is eligible to show, labels and device IDs are available. Before grant, `enumerateDevices()` returns one entry with empty `deviceId` and no label; the gate (`length > 1`) keeps the chevron hidden, so the user sees no broken-state UI.
|
||||
|
||||
**Voice tile text-selection suppression.** `VoiceUser` and `StreamTile` outer containers carry `data-context-menu` attribute. The global `@media (max-width: 767px)` rule in `globals.css` applies `user-select: none` (and inheriting `-webkit-touch-callout: none` via the `*` rule) to every `[data-context-menu]` element, which also inherits to children. Without this, iOS Safari's long-press handler synthesizes the context menu correctly via `useGlobalLongPress`, but the OS *also* triggers native text selection during the 500 ms hold, leaving the entire page's text highlighted in the background after the menu opens. The `data-context-menu` opt-in is the established convention used by `Message.tsx`, `MobileDmsScreen.tsx`, `MobileFolderSheet.tsx`, and `MobileSpacesScreen.tsx`.
|
||||
|
||||
**Disconnect:** Same logic as mini-bar (handles DM calls and space voice, calls `disconnectFn`, pops screen).
|
||||
|
||||
**Guard:** If `currentVoiceChannelId` is falsy, calls `popMobileScreen()` and returns null.
|
||||
|
||||
**Layout sizing.** The grid body is `flex-1` between the 48 px header and the floating control bar; on a 390 × 844 viewport the body is roughly 390 × 720, so:
|
||||
|
||||
- **1 participant:** the lone tile fills the body minus padding (~366 × 206 at 16 : 9). Just the local user's avatar / camera with the standard speaking-ring + name overlay.
|
||||
- **2 participants:** `useGridLayout` picks the configuration that maximises tile area. On a portrait phone that's `cols=1, rows=2` — two stacked 16 : 9 tiles ~366 × 200 each, vertically centred.
|
||||
- **3–4 participants:** typically `cols=2, rows=2` (4) or `cols=1, rows=3` (3). The same area-maximising algorithm runs on mobile and desktop; nothing is mobile-specific.
|
||||
- **Focused mode:** the focused tile fills the body minus the 120 px (max 20 vh) bottom strip; the strip horizontally scrolls if other-participant count exceeds the visible width.
|
||||
|
||||
No mobile-specific min-tile clamp exists. If the area-maximising solver picks an absurdly small tile for many participants, paginate-or-scroll is intentionally not added — the issue is symmetric with desktop and any future mobile-only paginator should land on desktop too.
|
||||
|
||||
---
|
||||
|
||||
## MobileFolderSheet
|
||||
@@ -645,6 +670,7 @@ keyboardOcclusion = window.innerHeight - (visualViewport.offsetTop + visualViewp
|
||||
It returns `{ value, keyboardOpen, height, offsetTop }`:
|
||||
- `value` — `'<n>px'` when the keyboard is open (the occlusion), or the literal `'env(safe-area-inset-bottom)'` string when it is not. Provided for legacy / fallback use.
|
||||
- `keyboardOpen` — `true` when `keyboardOcclusion > 1`.
|
||||
- `textInputFocused` — `true` while a text-entry element holds focus. Required for iOS PWA standalone where iOS itself shrinks the layout viewport for the keyboard, so `vv.height === innerHeight` and `keyboardOpen` stays `false` even though the keyboard IS up. Consumers OR `keyboardOpen || textInputFocused` to detect "keyboard probably open". The state-equality check inside the hook MUST include this field — historically it was missing, so on iOS PWA the hook silently dropped focus changes; the composer's `bottom` style stayed pinned to `env(safe-area-inset-bottom) + 6px` while the keyboard was open, the `--composer-clearance` ResizeObserver effect's deps fired stale values, and on close the last message overlapped the composer's top edge by ~4 px.
|
||||
- `height` — live `visualViewport.height` in pixels (or `null` if `visualViewport` is unavailable).
|
||||
- `offsetTop` — live `visualViewport.offsetTop` in pixels.
|
||||
|
||||
|
||||
+46
-1
@@ -17,6 +17,26 @@ Source files:
|
||||
5. Client calls `POST /api/livekit/token { channelId }` → gets JWT + LiveKit URL
|
||||
6. Client connects to LiveKit room with token
|
||||
|
||||
### Microphone pre-arm (iOS user-gesture discipline)
|
||||
|
||||
`utils/voice.joinVoiceChannel` fires `AudioContext.resume()` and `AudioManager.setInputDevice(inputDeviceId)` (which ends in `getUserMedia({audio:…})`) **synchronously inside** the click handler, before the `connectFn(channelId)` call. iOS Safari only surfaces the microphone permission prompt when `getUserMedia` is invoked from inside an active user-gesture; the original flow only acquired the mic in `useLiveKit`'s `syncMic` effect, which fires AFTER `room.connect()` resolves (token fetch + WS handshake) — many awaits past the gesture window. iOS PWA standalone is especially strict and would silently never surface the prompt; the user would see "Waiting for others to join…" indefinitely until they locked/unlocked the device (which iOS treats as a fresh activation).
|
||||
|
||||
Pre-arm is fire-and-forget: the mic acquisition runs in parallel with the LiveKit handshake, and `AudioManager.inputSwitchChain`'s serialization guarantees `useLiveKit.syncMic`'s subsequent call short-circuits on the already-acquired `currentStream` (no double prompt, no second `getUserMedia`).
|
||||
|
||||
### Listener mode (`micPermissionDenied`)
|
||||
|
||||
When the user denies the prompt (or has previously denied at the OS level), the pre-arm's `setInputDevice` rejects with `NotAllowedError`. The `voiceStore.micPermissionDenied` flag is set to `true` and the LiveKit connect proceeds anyway — the user appears in the voice channel as a connected participant who can hear others but has no microphone publication. `useLiveKit.syncMic` checks the flag at the top of its body and skips the publish branch entirely.
|
||||
|
||||
The flag clears via:
|
||||
- `requestMicPermission()` (in `utils/voice.ts`) — must be called from a user-gesture handler (button click). Clears `AudioManager.inputDenialError` cache, calls `setInputDevice` from a fresh activation. On success, sets `micPermissionDenied=false` and `useLiveKit.syncMic` re-fires (dep on `micPermissionDenied`) to publish the freshly acquired track.
|
||||
- `voiceStore.leaveVoice()` / `handleForceDisconnect()` / `resetSession()` / `reset()` — flag resets so the next join attempts a fresh prompt.
|
||||
|
||||
UI affordances:
|
||||
- **Mobile (`MobileVoiceFullScreen`):** banner below the header reads "Microphone access denied — You're listening only". A right-aligned "Allow microphone" button calls `requestMicPermission()`.
|
||||
- **Desktop (`VoiceControlBar`):** *(Future)* — same listener-mode state needs a parity affordance. Desktop is unaffected by the iOS gesture-window bug in practice (browsers there prompt on `getUserMedia` regardless of activation state), but if a desktop user denies the prompt, the same flow applies.
|
||||
|
||||
`AudioManager.inputDenialError` caches the most recent `NotAllowedError`. Subsequent `setInputDevice` calls re-throw the cached error rather than firing a second `getUserMedia` — iOS otherwise would queue a second prompt that has lost its activation, leading to a silent hang. Cleared by `AudioManager.clearInputDenial()` (called from `joinVoiceChannel`'s pre-arm and from `requestMicPermission`).
|
||||
|
||||
**Token grants (space channels):**
|
||||
- SPEAK → can publish MICROPHONE + CAMERA
|
||||
- STREAM → can publish SCREEN_SHARE + SCREEN_SHARE_AUDIO
|
||||
@@ -249,9 +269,34 @@ The "Share system audio" toggle in `ScreenSharePicker` adds an audio track to th
|
||||
|
||||
---
|
||||
|
||||
## Mobile Voice Rendering
|
||||
|
||||
Mobile (`MobileVoiceFullScreen`) renders the **same** `VoiceGrid` component as desktop. There is no mobile-specific tile component — the rendering, attach/detach, adaptive-stream subscription, focused-publisher layout, and context menus all come from the shared `VoiceGrid` / `VoiceUser` / `StreamTile` pipeline. The only mobile-specific addition is auto-focus on the first live screen-share publication (so phone users don't have to discover tap-to-focus).
|
||||
|
||||
See `docs/systems/mobile-ui.md` → "MobileVoiceFullScreen" for the auto-focus state machine, control-bar wiring, and layout sizing.
|
||||
|
||||
**Why the shared component path matters.**
|
||||
|
||||
- Local camera preview: `VoiceUser` attaches the local participant's `videoTrack` to a `<video muted>`. Mobile gets the self-preview "for free".
|
||||
- Remote cameras: `Track.attach(videoEl)` registers the element with LiveKit's `RemoteVideoTrack` adaptive-stream observer, so the SFU automatically picks the appropriate simulcast layer based on the painted tile size on the phone. No mobile-specific bitrate clamp is needed.
|
||||
- Screen-share: `StreamTile` lazily subscribes via `setStreamSubscription` only after the user taps "Watch Stream" (or auto-focus does so on mobile, which currently still requires the user to tap the in-tile "Watch Stream" CTA — auto-focus only sets the focused publisher; it does not auto-subscribe to bandwidth-heavy screen-share tracks).
|
||||
- Mute / deafen / speaking-ring overlays, watch/unwatch controls, local mute, volume sliders — identical between mobile and desktop.
|
||||
|
||||
**Screen-share button wiring on mobile.** `MobileVoiceFullScreen`'s screen-share button calls `handleScreenShareAction()` from `utils/voiceActions`, **not** `voiceStore.toggleScreenShare`. The store action only flips the `isScreenSharing` boolean and never calls `getDisplayMedia`. The canonical `handleScreenShareAction` is shared with desktop's `VoiceControlBar` and the keybind manager; it calls `startScreenShare(room)` / `stopScreenShare(room)` and broadcasts voice status to peers. iOS Safari does not support `getDisplayMedia` (the call rejects); this is a platform limitation. Android Chrome supports it and works.
|
||||
|
||||
---
|
||||
|
||||
## Voice Fullscreen
|
||||
|
||||
The fullscreen toggle in `VoiceControlBar` flips the `voiceFullscreen` flag in `uiStore`; an effect in `MainContent.tsx` calls `voiceContainerRef.current.requestFullscreen()` (and exits via `document.exitFullscreen()` when the flag clears). A second effect listens to `fullscreenchange` and reflects the actual `document.fullscreenElement` back into the store, so pressing Esc or system-level fullscreen-exit keeps state in sync. `voiceChatOpen && !voiceFullscreen` hides the side chat panel while fullscreen is active.
|
||||
The fullscreen toggle in `VoiceControlBar` flips the `voiceFullscreen` flag in `uiStore`; an effect in `MainContent.tsx` enters/exits the browser's Fullscreen API on `voiceContainerRef`. A second effect listens to `fullscreenchange` and reflects the actual document fullscreen element back into the store, so pressing Esc or system-level fullscreen-exit keeps state in sync. `voiceChatOpen && !voiceFullscreen` hides the side chat panel while fullscreen is active.
|
||||
|
||||
**Cross-browser API fallback.** iOS Safari (and iPadOS pre-16.4) does not implement the standard `Element.requestFullscreen()` on generic elements, so the enter-fullscreen effect probes for the API in this order:
|
||||
|
||||
1. `el.requestFullscreen()` — standard
|
||||
2. `el.webkitRequestFullscreen()` — older WebKit (some iPads, older Safari)
|
||||
3. Silent fall-through — pure iPhone Safari has neither API on a `<div>` (only `HTMLVideoElement.webkitEnterFullscreen()` works, which we cannot use for the multi-tile voice container)
|
||||
|
||||
When neither native API is available the effect returns without throwing; the `voiceFullscreen` flag still applies `h-screen` to `voiceContainerRef`, which acts as the in-page maximize fallback (chat panel hides, header fades, control bar stays). The exit path mirrors this with `document.exitFullscreen()` → `document.webkitExitFullscreen()` → no-op. Both paths are wrapped in try/catch so a Promise rejection (e.g. user cancels via Esc mid-transition) does not surface as an unhandled error. The `fullscreenchange` listener is registered for both `fullscreenchange` and `webkitfullscreenchange`. Before this fallback, calling the missing API directly threw `TypeError: requestFullscreen is not a function` on iPhone Safari, which surfaced as a full-screen error overlay when an iPhone user crossed the 768 px desktop breakpoint in landscape mode.
|
||||
|
||||
**Overlay portals:** While fullscreen is active the browser's Fullscreen API renders only descendants of `voiceContainerRef`. Every overlay reachable during a call (context menus on `StreamTile`/`VoiceUser`/`VoiceChannel`, tooltips on the control bar, `ConnectionInfoPopover`, `ScreenShareSettingsPopover`, `ConfirmDialog` invoked from voice context-menu actions, and `ScreenSharePicker`) portals through `usePortalContainer()` so it lands inside the fullscreen element. Adding new overlays that can be opened from inside the call must follow the same contract — see `docs/systems/design-system.md` Surface Material Tiers.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user