AI agents are service participants. They join the room the same way humans do.
Four architecture patterns. Honest latency numbers. BYO every AI provider.
Core concepts
Three primitives power every AI integration.
Workers subscribe to room audio/video tracks, process frames, and publish results back — via the same signaling protocol as human participants.
Signaling-only room client for server-side AI workers. Available in Node.js (TypeScript) and Python. Subscribes to audio/video tracks, emits structured room events.
Server-side process managing worker lifecycle inside a room. Receives spawn requests from the Admin API, tracks health, enforces the per-license AI worker cap.
Extensibility interface workers implement. Provided adapters: TranscriptionAdapter, ModerationAdapter, AssistantAdapter, AnalyticsAdapter.
Pattern 1
Real-Time Transcription
Live captions, transcript export, meeting notes. Latency: 200–400ms (Deepgram Streaming) · 600–1500ms (Whisper-large local).
Host grants AI worker permission
Admin API spawns worker
POST /api/rooms/:roomId/ai-workers → { type: "transcription", provider: "deepgram" }
Worker subscribes to all publisher audio tracks via WorkerRoomClient
Audio frames stream to STT provider
Deepgram, Whisper, Azure Speech, etc.
Word-level chunks published as structured chat messages
{ role: "transcript", speakerId, text, startMs, endMs }
Browser SDK receives events and renders captions
Pattern 2
Meeting Assistant (LLM-Backed)
AI assistant answering questions in chat, summarizing discussion, taking action items. Works best as assistive/async — not sub-second voice conversation.
Transcription worker feeds a rolling context window
AssistantAdapter worker subscribes to room chat messages
Participant directs question to assistant
Worker sends rolling transcript + question to LLM API (OpenAI, Anthropic, Ollama, etc.)
LLM response published to room chat
Response streaming supported — partial tokens published as chat updates
Pattern 3
Real-Time Moderation
Audio policy enforcement, visual frame analysis, automated content moderation. Sample video at 1–2 fps on GPU instances; calibrate false positive rates carefully.
ModerationAdapter worker subscribes to all publisher tracks
Audio/video frames sent to moderation API
AWS Comprehend, Azure Content Safety, or local inference
On policy violation, worker calls Admin API
Mute or kick participant via adminClient.muteParticipant()
Moderation event emitted to audit log
Pattern 4
Analytics & Quality Coaching
Talk-time tracking, sentiment analysis, sales call scoring, speaker coaching. The lowest-latency AI path — processing runs on a slight delay without impacting call quality.
AnalyticsAdapter worker subscribes to audio without publishing back to room
Tracks per-speaker talk time, interruption events, pause patterns
Optionally streams sentiment scores to side-channel analytics backend
End-of-call summary stored to database adapter
Optionally published as final room chat message
BYO Provider
TomatoRTC does not bundle, proxy, or bill for AI provider APIs.
Workers are your code, in your infrastructure, calling your provider accounts.
Browser-side
Client AI runs entirely in the browser.
@rtc-sdk/client-ai — UI-side AI features that don't require server processing. No audio or video leaves the client device.
VAD (Voice Activity Detection)
Detects local speech onset/offset without sending audio to a server. Used for UI indicators and local recording gating.
Ambient Noise Detection
Classifies background noise level to prompt users to mute or improve microphone placement.
WebGL Frame Preprocessing
GPU-backed frame downscaling before sending frames to MediaPipe or remote APIs.
Latency budget
Honest numbers from capture to voice response.
For sub-500ms voice AI, specialized real-time inference infrastructure (Groq, Cerebras, local GPU) is required for STT and LLM stages.
Scaling
Multiple worker types can run simultaneously in one room.
Each transcription worker streaming to cloud STT consumes ~0.5 vCPU. Local Whisper inference requires 1–4 vCPU or GPU per concurrent stream.
Concurrent AI workers across all rooms
Concurrent AI workers across all rooms
Unlimited (fair-use reviewed)
Division of responsibility
Clear boundaries between platform and your code.
TomatoRTC ships the primitives. You wire in the providers, business logic, and infrastructure.
WorkerRoomClient · AiWorkerSupervisor · WorkerAdapter interfaces (Transcription, Moderation, Assistant, Analytics) · Admin API for worker spawn/teardown · Chat message system for publishing results
STT, LLM, TTS provider accounts and API keys · Worker business logic (implementing WorkerAdapter) · Infrastructure to run workers (containerized Node.js or Python) · Any custom AI models or fine-tunes
Ready to build a voice AI pipeline?
- 1 Review the architecture patterns
Pick the one that matches your use case — transcription, assistant, moderation, or analytics.
- 2 Read the worker development guide
WorkerAdapter interface, supervisor API, and typed WorkerOutput — full guide in the monorepo.
- 3 Run the kitchen-sink AI demo
A working transcription worker ships as a runnable example in the monorepo.