ADR 0006: Realtime Engine — OpenAI Realtime for Launch

Accepted

Status

Accepted

2026-07-17

Two realtime architectures exist in the repo:

Own-backend streaming (realtime-architecture.mdx): audio-capture → audio-engine (AudioWorklet, client VAD, Opus) → WebSocket → backend ASR → LLM → suggestions. Clean layering, sub-1s latency budget.
OpenAI Realtime WebRTC (implemented in apps/desktop/src/lib/live-call.ts + cosella-api): the API mints an ephemeral client secret with the org’s knowledge base and prospect intel injected into the session instructions; the client adds the system-audio track to a WebRTC connection straight to OpenAI. Server-side VAD, transcription, and suggestion generation all happen in one hop. This is the architecture the Lightforth copilot runs in production.

Decisive constraints:

Architecture 1 requires a backend streaming ASR + LLM pipeline that does not exist in cosella-api. Building and operating it (streaming ASR vendor, session fan-in, prompt orchestration) is weeks of work and a new cost center, to reach latency OpenAI Realtime already delivers.
Client-side VAD is a documented production incident on the OpenAI path: a client once overrode the server VAD config and reset the noise threshold, re-enabling phantom “thank you” commits from room noise. The minted session config (threshold 0.75, silence 1100ms, gpt-4o-mini-transcribe) is the single source of truth; nothing client-side may gate or re-detect turns.
ElectronCaptureManager.acquireDeviceAudio() depends on Chromium loopback via setDisplayMediaRequestHandler. Loopback audio works on Windows with our pinned Electron 33; the macOS loopback path needs a newer Electron than the pin (the code comment says 39). The handler also wasn’t registered in the desktop main process until now.

Engine: OpenAI Realtime WebRTC for launch. The desktop live call keeps the implemented engine (live-call.ts): session minted by cosella-api, system audio only (never mic), server VAD as single source of truth, yes-check + rescue assess per customer turn.
@cosella/audio-capture is the shared capture layer. The desktop main process now registers setDisplayMediaRequestHandler with audio: 'loopback', and the engine tries acquireDeviceAudio() first. Acquisition order: cached picker stream → shared manager (loopback) → macOS native tap helper (ScreenCaptureKit binaries, production-proven) → last-ditch picker. When Electron is upgraded to a loopback-capable version on macOS and a live-call test passes, the native tap becomes dead code and can be removed.
@cosella/audio-engine (worklet/VAD/Opus) is deferred, not deleted. It is the right shape for a future own-backend streaming pipeline (vendor independence, cost control at scale), but it must not sit in the OpenAI path — no client VAD, no re-encoding of the WebRTC track.

One capture API for web and desktop; the platform quirks live behind it.
Launch requires no new backend infrastructure for the copilot.
Revisit after launch: if OpenAI Realtime cost or vendor risk becomes material, architecture 1 is the successor and audio-engine is its ready-made client half. That migration only changes what consumes the captured MediaStream.

The versioned spec for the own-backend streaming architecture lives in the repo:

Design overview: engineering/realtime-architecture (this docs site)
Full plan incl. binary frame protocol: packages/realtime/docs/superpowers/plans/2026-07-17-realtime-audio-streaming.md

Changes to the wire protocol go into those documents first; this ADR only records the decision to defer.