Audio Engine

Audio Engine

@cosella/audio-engine takes raw MediaStream objects from audio-capture and produces Opus-encoded audio frames with VAD annotations, ready for transport. It runs entirely on the Web Audio API — platform-agnostic across Electron and browsers.

Installation

pnpm add @cosella/audio-engine

Quick Start

import { AudioPipelineImpl, createOpusEncoder } from '@cosella/audio-engine';
import type { DEFAULT_AUDIO_CONTRACT } from '@cosella/domain';

// Create encoder (async — checks WebCodecs support at startup)
const createEncoder = await createEncoderFactory(DEFAULT_AUDIO_CONTRACT);

// Create pipeline
const pipeline = new AudioPipelineImpl(createEncoder);

// Attach a MediaStream (from audio-capture)
await pipeline.attach('mic', micStream);
await pipeline.attach('loopback', loopbackStream);

// Listen for encoded frames
pipeline.on('frame', ({ channel, seq, captureMs, opus }) => {
  // Send to server via realtime transport
});

pipeline.on('vad', ({ channel, speaking, timestampMs }) => {
  // Notify server of speech boundaries
});

pipeline.on('level', ({ channel, rms }) => {
  // Update UI level meters (0-1)
});

// Cleanup
pipeline.dispose();

Architecture

MediaStream (48kHz from browser)
    │
    ▼
AudioContext({ sampleRate: 16000 })    ← browser resamples
    │
    ▼
AudioWorkletProcessor                  ← audio render thread (off main thread)
    │  • Accumulates 20ms frames (320 samples)
    │  • Stamps captureMs (session-relative)
    │  • Runs VAD per frame
    │  • Posts to main thread (zero-copy transfer)
    ▼
AudioPipeline (main thread)
    │  • Encodes PCM → Opus (async via WebCodecs)
    │  • Emits frame, vad, and level events
    ▼
Consumer (realtime transport)

Components

AudioPipeline

The orchestrator that wires everything together. One pipeline handles multiple channels, each with its own encoder and VAD instance.

interface AudioPipeline {
  attach(channel: CaptureChannel, stream: MediaStream): Promise<void>;
  detach(channel: CaptureChannel): void;
  on<K extends keyof PipelineEvents>(event: K, handler: PipelineEvents[K]): () => void;
  dispose(): void;
}

VadDetector

Energy + zero-crossing rate voice activity detection with adaptive noise calibration.

import { VadDetector } from '@cosella/audio-engine';

const vad = new VadDetector();

// Process a 20ms frame (320 samples at 16kHz)
const { speaking, rms } = vad.process(pcmFloat32);

Parameter	Value
Calibration	First 500ms of ambient noise
Speech onset	3 voiced frames (60ms)
Speech offset	15 silent frames (300ms)
ZCR range	0.02 – 0.25

Opus Encoder

Two-tier encoding with automatic fallback:

1. WebCodecs AudioEncoder — hardware-accelerated, checked via isConfigSupported() at startup
2. NoopEncoder — passthrough for testing (libopus.js WASM planned as production fallback)

import { createEncoderFactory } from '@cosella/audio-engine';

const createEncoder = await createEncoderFactory(DEFAULT_AUDIO_CONTRACT);
const encoder = createEncoder(); // One per channel

Sequence Numbers

seq is a time-position counter: it increments by 1 for every 20ms of capture time, regardless of whether a frame was emitted. Silence-skipped and backpressure-dropped frames leave gaps. The server reconstructs silence from gaps.

Testing

The pipeline exposes handleWorkletMessage(channel, data) for testing without a browser environment:

import { AudioPipelineImpl, NoopEncoder } from '@cosella/audio-engine';

const pipeline = new AudioPipelineImpl(() => new NoopEncoder());

pipeline.on('frame', (data) => {
  expect(data.seq).toBe(0);
});

pipeline.handleWorkletMessage('mic', {
  pcm: new Float32Array(320).buffer,
  captureMs: 0,
});