Real-Time Audio Streaming Architecture

Cosella’s core value proposition is live call coaching — AI-generated suggestions that appear while the rep is still on the phone. This requires capturing audio from both the rep’s microphone and the prospect’s voice (via device/system audio), streaming it to a server for transcription, and displaying AI suggestions within one second of speech.

System Overview

┌──────────────────────────────────────────────────────────────┐
│  App Layer (desktop / dashboard)                             │
│  useCallSession — orchestrates the full call lifecycle       │
│                                                              │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────┐  │
│  │ audio-capture│→ │ audio-engine │→ │ realtime           │  │
│  │              │  │              │  │                    │  │
│  │ Mic stream   │  │ AudioWorklet │  │ WebSocket (binary  │  │
│  │ Device audio │  │ VAD          │  │   + text frames)   │  │
│  │              │  │ Opus encoder │  │ Auth, reconnect    │  │
│  └─────────────┘  └──────────────┘  │ Send queue         │  │
│                                      └────────┬───────────┘  │
└───────────────────────────────────────────────┼──────────────┘
                                                │
                                     WebSocket  │  binary audio ↑
                                     connection │  text events  ↓
                                                │
                                      ┌─────────▼──────────┐
                                      │  Backend Server     │
                                      │  ASR → LLM →        │
                                      │  Suggestions        │
                                      └────────────────────┘

Latency Budget

The target is sub-1 second from speech to the first displayed suggestion token:

Stage	Budget	Owner
Audio capture + encode	~25ms	Client
Transport to server	~50ms	Network
Streaming ASR (partial)	~200ms	Server
LLM first token	~300ms	Server
Transport back	~50ms	Network
Render	~10ms	Client
Total	~635ms

This works because the architecture is streaming at every layer — no batching, no waiting for complete utterances.

Package Architecture

Four packages, layered bottom-up:

domain          ← protocol types, audio contract, binary frame constants
    ↑
audio-capture   ← platform-specific MediaStream acquisition
    ↑
audio-engine    ← AudioWorklet + Opus encoding + VAD (platform-agnostic)
    ↑
realtime        ← WebSocket transport with binary + text multiplexing

The only platform-specific code is in audio-capture — how you get a MediaStream. Everything downstream is identical on Electron and web.

Data Flow

Mic (getUserMedia)  ──→  AudioWorklet  ──→  Opus Encoder  ──→  WebSocket
                         (16kHz, 20ms       (WebCodecs or       (7-byte header
Device Audio ───────→     frames, VAD,       libopus.js          + opus payload)
(Electron loopback       level metering)     fallback)
 or getDisplayMedia)

Wire Protocol

A single multiplexed WebSocket carries both binary audio frames (upstream) and JSON text events (bidirectional).

Binary Audio Frames (client → server)

Each 20ms Opus frame is sent as a binary WebSocket message with a 7-byte big-endian header:

Bytes	Field	Type	Description
0	Channel	uint8	`0x01` = mic (rep), `0x02` = loopback (prospect)
1-4	Sequence	uint32	Time-position: increments every 20ms regardless of emission
5-6	Delta	int16	Signed jitter correction (ms offset from expected time)
7+	Payload	bytes	Opus-encoded audio (~40-80 bytes)

Sequence gaps signal dropped or skipped frames. The server reconstructs silence from gaps and never requests replay.

Text Events (JSON)

Client → Server:

call.start — begins a session with audio contract, consent, and LLM context
call.end — ends the session
call.resume — reconnects with last sequence numbers per channel
vad.speech_start / vad.speech_end — client-side speech boundary signals

Server → Client:

transcript.partial — streaming ASR result (~200ms updates)
transcript.final — finalized segment with confidence score
suggestion.partial — streaming LLM token (for typewriter display)
suggestion.final — complete suggestion
suggestion.aborted — superseded by newer utterance
call.ready — server has warmed ASR/LLM context, client may begin audio
ping — liveness heartbeat (every 15s if no other message)

Connection Lifecycle

WebSocket open
    │
    ├── auth { token }               ← client authenticates (5s timeout)
    │
    ├── auth.ok                      ← server confirms
    │
    ├── call.start { ... }           ← client declares audio contract + consent
    │
    ├── call.ready { epochMs }       ← server ready, audio may begin (5s timeout)
    │
    ├── Binary audio frames ──────►  streaming at 50 frames/sec
    ├── vad events ────────────────► speech boundaries
    │
    │◄── transcript.partial          ~200ms after speech
    │◄── transcript.final            on utterance end
    │◄── suggestion.partial          streaming tokens (sub-1s target)
    │◄── suggestion.final / aborted
    │
    ├── call.end { reason }          ← client ends session
    └── close

Audio Capture

Desktop (Electron)

The desktop app captures both mic and system audio simultaneously:

Mic: Standard getUserMedia() — identical to web
System audio: Chromium’s built-in loopback via setDisplayMediaRequestHandler with audio: 'loopback'. Uses WASAPI on Windows, CoreAudio on macOS. Auto-approved by the main process — no user picker.

Electron 39.x is pinned. Regressions reported on 40.x for macOS desktop audio. Non-silent loopback is a required smoke test before any Electron upgrade.

Web Browser

Mic: Standard getUserMedia()
Device audio: getDisplayMedia() with the screen share picker. User must check “Share audio.”

Web platform ceiling: System audio capture is effectively Windows + Chrome only. On macOS, the browser can only share tab audio, not desktop app calls. This is a browser limitation — the Chrome extension (v2) will address it.

v1 Assumption

Headphones required. The rep must use headphones so the mic doesn’t pick up prospect audio from speakers. This avoids echo cancellation complexity for v1.

Audio Engine

AudioWorklet Processing

Audio is processed on the audio render thread (not the main thread) via an AudioWorkletProcessor:

Receives 128-sample render quanta from the browser
Accumulates into 20ms frames (320 samples at 16kHz)
Stamps each frame with captureMs (session-relative monotonic time)
Runs VAD (voice activity detection) per frame
Computes RMS level for UI meters
Posts frame + metadata to the main thread via MessagePort (zero-copy transfer)

Resampling

Browsers deliver audio at their native rate (typically 48kHz). The engine requests 16kHz via new AudioContext({ sampleRate: 16000 }). If the browser ignores the hint, manual resampling with anti-aliasing handles both 48kHz (3:1 integer ratio) and 44.1kHz (non-integer ratio).

Voice Activity Detection (VAD)

Client-side VAD tells the server when speech starts and stops, enabling:

ASR optimization: process only voiced segments
Suggestion timing: detect utterance boundaries for generation triggers
Bandwidth savings: Opus DTX during silence

Algorithm: energy + zero-crossing rate with adaptive calibration.

Parameter	Value	Purpose
Calibration window	500ms (25 frames)	Learn ambient noise level
Energy threshold	median × 3.0	Distinguish speech from noise
ZCR range	0.02 – 0.25	Speech has moderate zero-crossing rate
Onset hysteresis	3 frames (60ms)	Prevent false triggers
Offset hysteresis	15 frames (300ms)	Don’t cut off mid-pause

Known limitation: Energy-based VAD treats hold music as speech. Silero-VAD WASM is the v2 upgrade path.

Opus Encoding

Two encoder paths with automatic fallback:

WebCodecs AudioEncoder (preferred) — hardware-accelerated, low CPU. Checked at startup via AudioEncoder.isConfigSupported().
libopus.js WASM (fallback) — works everywhere, ~3× more CPU.

Each channel gets its own encoder instance to prevent interleaved audio corruption.

Transport Reliability

Backpressure

An app-level send queue (25 frames / ~500ms per channel) sits between the audio pipeline and the WebSocket:

Drop policy: drop oldest when full (most recent audio is most useful for ASR)
Flush gate: only send when ws.bufferedAmount < 16KB
Degradation signal: rolling dropped-frame rate over 5 seconds drives connectionQuality:

Drop Rate	Quality	UI
< 5%	`good`	No indicator
5-20%	`degraded`	Yellow indicator
> 20%	`poor`	”Poor connection — coaching may lag”

Dropped loopback frames are the prospect’s voice — this directly degrades transcription and suggestion quality. Shedding is a quality tradeoff, not free.

Reconnection

On unexpected disconnect:

Exponential backoff with ±50% jitter (150ms base → 30s cap, 8 retries max)
Fresh getToken() per attempt (handles expired tokens)
call.resume with last sequence numbers per channel
Server responds call.ready (session preserved) or SESSION_NOT_FOUND (fall back to fresh call.start)
Client re-sends current VAD state and re-anchors timestamps after resume

Liveness

Server sends an application-level ping if no message in 15 seconds. Client tracks lastMessageReceived — if 30 seconds elapse with no message (2 missed pings), force-close and reconnect. Browser WebSocket API doesn’t expose protocol-level ping/pong, hence application-level heartbeats.

The protocol includes consent fields from day one:

repConsented — rep acknowledged recording
prospectNotified — rep confirmed prospect was notified
jurisdiction — optional, for compliance audit

v1 server behavior: proceed and log for all combinations. The consent state is recorded with the call record. Future versions may block call.start based on jurisdiction requirements.

Recording/transcribing calls requires consent in many jurisdictions (two-party consent in several US states, GDPR in the EU). The consent prompt is surfaced during call setup.

Error Recovery

Core principle: mic-only is always a valid degraded state.

Error	Response	User sees
Mic permission denied	Session fails	”Microphone access required”
Device audio not shared	Continue mic-only	”Device audio unavailable — mic only”
Device audio track ended	Detach, continue mic	”Device audio stopped”
Auth token expired	Refresh + retry once	Silent if refresh succeeds
Auth token invalid	Session fails	”Please sign in again”
Connection lost	Reconnect with resume	”Reconnecting…”
Max retries exhausted	Session fails	”Connection lost”
Encoder can’t keep up	Skip frames	Silent (observability)

Future Enhancements

Chrome extension for tab audio capture (eliminates screen share picker on web)
Silero-VAD WASM for ML-based speech detection
Native Electron addon for per-process audio capture
Local transcript persistence for offline resilience
Echo cancellation for speaker mode (no headphones)

Real-Time Audio Streaming Architecture

Real-Time Audio Streaming Architecture

System Overview

Latency Budget

Package Architecture

Data Flow

Wire Protocol

Binary Audio Frames (client → server)

Text Events (JSON)

Connection Lifecycle

Audio Capture

Desktop (Electron)

Web Browser

v1 Assumption

Audio Engine

AudioWorklet Processing

Resampling

Voice Activity Detection (VAD)

Opus Encoding

Transport Reliability

Backpressure

Reconnection

Liveness

Consent

Error Recovery

Future Enhancements