TomatoRTC

Voice agents as room citizens — not sideband hacks.

Workers join with role: "service", subscribe to audio inside the SFU path, run your STT/LLM/TTS, and publish back — provider choice and data boundaries stay with you.

Service participants BYO providers Supervisor API Client perception
  1. Human participants publish live audio into the room over mesh or SFU paths.
  2. A service worker joins the same room via WorkerRoomClient with role service.
  3. The worker subscribes to remote audio, runs VAD turn detection, then STT, LLM, and TTS adapters.
  4. Structured WorkerOutput events and optional synthesized audio publish back into the room.
Human audio in room
Worker adapters
Structured output

The gap

Most "voice AI" stacks bolt agents on beside the call.

LiveKit Agents, Daily Pipecat, and OpenAI Realtime API often treat AI as a parallel pipeline — separate WebSockets, separate billing, separate failure modes. Product teams lose room context, struggle with compliance, and cannot swap STT/LLM vendors without rewriting orchestration.

TomatoRTC approach

Workers are first-class room members. They see the same participants, tracks, chat, and permissions model as humans — with typed WorkerOutput events any client can consume.

Room-native pattern

  • Worker subscribes inside SFU/mesh boundary
  • Output on room chat metadata + optional audio publish-back
  • BYO Deepgram, Whisper, OpenAI, Ollama — your contracts

Sideband pattern

  • Audio forked to third-party realtime API
  • Agent state outside room protocol
  • Harder audit and tenant isolation story

Service participants

Same room model — different role.

A Node.js or Python process joins via WorkerRoomClient, plugs in a WorkerAdapter, and implements only the hooks it needs: processTrack for audio taps, processMessage for chat-triggered flows, participantJoined for greetings. Swap adapters to change providers without touching subscription wiring.

Transcription

Stream remote audio to Whisper, Deepgram, or custom ASR.

Assistant

LLM replies as WorkerOutput — wire Ollama, Anthropic, OpenAI.

Moderation

Chat and transcript policy adapters with room events.

Custom pipeline

Analytics, translation, compliance — any WorkerKind.

Note
Note Slide context
Worker STT/LLM/TTS are customer/provider integrations — TomatoRTC ships subscription, output protocol, and supervisor. No bundled foundation model or inference platform. See the workers guide for integration details.

Four integration surfaces

From browser perception to server-side room output.

Browser VAD + perception

@rtc-sdk/client-ai — voice activity, optional browser STT, MediaPipe vision on supported platforms.

Per-room supervisor

AiWorkerSupervisor — spawn worker fleet via POST /api/rooms/:roomId/ai-workers.

Track processing

processTrack on remote audio — streaming STT from live room media.

Voice-out publish

Workers publish synthesized audio tracks — agent speaks in the room mix.

Client-side AI

Lightweight perception without bloating the core SDK.

@rtc-sdk/client-ai installs at runtime via client.use(TomatoAI()). Optional MediaPipe engine for landmarks/gestures; Web Audio VAD (Voice Activity Detection); browser SpeechRecognition wrapper where supported. Events normalize to a typed bus — publish low-volume events over a data channel when needed.

  • Keep ML optional core browser SDK stays lean; AI is a plugin package.
  • Capability probes kitchen-sink AI perception panel shows what this browser supports.
  • Self-host models production should host MediaPipe WASM/assets, not rely on CDN defaults.
  • Honest browser matrix STT/VAD varies; validate before customer commitments.

Operations

Spawn and stop workers per room — not one global bot.

Enable RTC_AI_WORKERS_SUPERVISOR=1 on the signaling server. Participants with room:ai:workers permission start or stop the worker fleet for their room. Workers idle-stop when the last human leaves — service participants do not keep fleets alive alone.

  • Admin console + REST same API surface for ops and in-room buttons.
  • Multi-topology safe PID dirs keyed by room + signaling host.
  • Output stream kitchen-sink Messages → AI & worker output panel.
Note
Note Slide context
Voice AI agents market ~$2.5B (2025) → ~$35B (2033), ~39% CAGR — Grand View Research. Directional sizing only. Supervisor requires repo checkout + configured providers (e.g. Ollama) on signaling host — see the workers guide.

Positioning

Flexible architecture — not a turnkey agent SaaS.

TomatoRTC wins when buyers need tenant isolation, audit trails, BYO AI providers, and room-native media boundaries. We do not ship bundled STT/TTS inference, telephony PSTN bridge, or a hosted agent marketplace — we ship the protocol and wiring so your team owns the product experience.

vs LiveKit Agents

Same service-participant ambition + tenant/audit/TURN built in.

vs OpenAI Realtime

Media stays on your SFU — no audio routed through third-party realtime API.

vs Pipecat

Room protocol owns output — not a separate orchestration framework lock-in.

Reference path

Wire Deepgram + OpenAI + worker publish-back — examples/node-bot, kitchen-sink.

Put a voice agent in your room — on your providers, your terms.

  1. Kitchen-sink AI panel

    Enable AI perception + spawn workers from Messages tab

  2. Read workers guide

    WorkerAdapter, supervisor API, and client-AI perception — in the monorepo docs.

  3. Commercial

    Sales deck · Voice & video foundation

Sales deckVoice & video
Own the AI boundary Own providers Own room context