by Alem Tuzlak on Mar 12, 2026.

Text-based chat is table stakes. The next wave of AI applications is conversational — real voice, real time, with the AI hearing you, thinking, and responding as naturally as a phone call. TanStack AI now ships first-class support for realtime voice conversations, with a clean provider-agnostic architecture, client-side tool execution, multimodal input, and a React hook that makes the whole thing feel like building a form.

Two providers are supported out of the box: OpenAI Realtime (via WebRTC) and ElevenLabs (via WebSocket). The adapter system means more providers can slot in without touching your application code.

Realtime voice chat has a fundamental constraint that text chat doesn't: the audio stream connects directly from the browser to the provider. You can't proxy a WebRTC session through your server. But you also can't ship your API key to the client.

TanStack AI solves this with a token/connection split:

1. Your server generates a short-lived ephemeral token using realtimeToken() — this never exposes your API key
2. Your client uses that token to establish a direct connection via a provider-specific adapter
3. Audio capture, playback, tool execution, and state management all happen client-side through useRealtimeChat

The token is the only thing that crosses your server boundary. Everything else is a direct browser-to-provider stream.

The server-side surface is intentionally tiny. Generate a token, return it to the client:

For ElevenLabs, swap the adapter:

That's the entire server-side implementation. The token adapters handle the provider-specific session creation (OpenAI's /v1/realtime/sessions endpoint, ElevenLabs' signed URL generation) and return a uniform RealtimeToken with token, expiresAt, and config. The client auto-refreshes tokens before they expire.

The React hook is where the magic happens. It manages the entire connection lifecycle, audio capture/playback, voice activity detection, tool execution, message state, and audio visualization — all behind a single hook:

That's a working voice chat. The hook requests microphone access on connect, streams audio to the provider, plays back the AI's response, tracks transcripts in real time, and tears everything down on disconnect or unmount.

The mode state gives you a single value that describes what's happening in the conversation right now:

This is enough to build responsive UI indicators — pulsing microphone icons, thinking spinners, speaking animations — without managing multiple boolean flags.

How does the system know when the user starts and stops talking? That's voice activity detection (VAD), and TanStack AI supports three modes:

Server VAD is the simplest — the provider handles everything. Semantic VAD (OpenAI only) uses the model's understanding of conversation flow to decide when the user is done speaking, controlled by the semanticEagerness option ('low', 'medium', 'high'). Manual mode gives you full control for push-to-talk UIs:

The realtime system uses TanStack AI's isomorphic toolDefinition() system. Define your tool once with Zod schemas, implement it with .client(), and pass it directly to useRealtimeChat:

Then pass them in:

When the AI decides to call a tool, the RealtimeClient executes it locally in the browser, sends the result back to the provider, and the AI continues talking with the tool's output. The user hears a natural response — "It's currently 72 degrees and sunny in San Francisco" — with no visible round-trip.

Multimodal Input: Text and Images Alongside Voice

Voice is the primary input, but you're not limited to it. sendText() lets users type when voice isn't practical, and sendImage() (OpenAI only) lets users share images for the AI to see and discuss:

The user can be mid-conversation by voice, snap a photo, send it, and ask "What's in this image?" — all within the same realtime session.

The hook exposes audio levels and raw frequency/time-domain data for building visualizations:

inputLevel and outputLevel update on every animation frame while connected — use them for simple volume bars:

For waveform or frequency visualizations, use the raw data getters with a canvas:

While the user or assistant is speaking, you get streaming transcription via pendingUserTranscript and pendingAssistantTranscript. These update in real time as speech is recognized and are cleared when the final message is committed to messages:

Final messages land in the messages array as RealtimeMessage objects with typed parts — text, audio (with transcript), image, tool-call, and tool-result:

Users can interrupt the AI mid-sentence — just start talking (with server/semantic VAD) or call interrupt() programmatically. The AI stops speaking, the interrupted message is marked with interrupted: true, and the conversation continues naturally. The onInterrupted callback fires so you can update UI state:

The full set of session options covers everything you need to tune the voice experience:

Because adapters are just objects, you can switch providers based on user choice without restructuring your code:

The server token endpoint picks the right adapter too:

If you're not using React — or need lower-level control — the RealtimeClient class from @tanstack/ai-client provides the same functionality without framework coupling:

This is how you'd integrate realtime voice into Solid, Vue, Svelte, or any other framework — wrap the RealtimeClient in your framework's reactivity primitives.

The realtime system gives you the building blocks. Here's what falls out of them:

Voice-controlled dashboards — "Show me last week's revenue" triggers a tool that queries your analytics API. The AI reads back the numbers while the dashboard updates in real time.

Multimodal customer support — users describe a problem by voice, send a screenshot, and the AI sees both. Tool calls look up their account, check order status, or file a ticket — all executed client-side.

Language tutoring — semantic VAD with low eagerness gives students time to think. The AI listens patiently, then responds with corrections. pendingUserTranscript shows what's being heard so students can self-correct.

Accessibility interfaces — voice in, voice out, with text transcripts for deaf-accessible screen reading. outputModalities: ['audio', 'text'] gives you both simultaneously.

Field data collection — a technician inspects equipment, describes what they see, sends photos, and the AI logs structured data via tool calls. Push-to-talk with vadMode: 'manual' works in noisy environments.

Set your API key in the environment:

The full documentation is available in the Realtime Voice Chat Guide.