How it works

The loop

microphone → speech recognition → knowledge retrieval → language model
           → text-to-speech → lip-synced video → WebRTC → screen

Every stage streams. The language model starts generating before the user has finished being transcribed; speech synthesis starts on the first complete sentence rather than the full answer; the renderer starts drawing as the first audio arrives. Nothing waits for the stage before it to finish completely.

Latency

Around two seconds voice-to-voice when the GPU is co-located with the user (same cloud region, or the same building for a kiosk). That is the number to plan around.

What moves it:

• Distance. A user in Riyadh talking to a GPU in Frankfurt pays the round trip twice.
• The language model. A large hosted model with a long system prompt is slower than a small self-hosted one. This is usually the biggest single lever.
• Knowledge retrieval. Searching a large document set adds time before the model can start.

Full-duplex speech

The avatar listens while it talks. When the user starts speaking, generation and playback stop mid-sentence and the avatar switches to listening — the same way a person would. This is what makes a kiosk feel usable in a noisy hall, and it is handled server-side with voice activity detection plus echo gating, so a plain microphone works.

Languages

Over 50 languages for both recognition and speech, including Arabic (with dialect handling), English, Russian, Mandarin, Spanish, French, Hindi, and Turkish. Auto-detection is available; for kiosks we usually pin the expected languages, which measurably improves recognition accuracy.

Two rendering paths

	2D photoreal	3D Unreal Engine 5
Look	Video-realistic, built from footage of a real person	Stylised or realistic CG character
Rendering	Neural lip-sync per frame, hardware-encoded, pushed over WebRTC	Real-time game-engine rendering
Runs on	Server GPU	Server GPU, or directly on the end-user device including phones
Best for	Maximum realism, brand ambassadors, public-sector spokespeople	Mascots, characters, animation freedom, low-cost scale-out

Both use the same voice pipeline, the same knowledge base, and the same APIs. Choosing one is a look-and-cost decision, not an architectural one — see Avatars and voices.

What runs where

The heavy work — recognition, the model, speech synthesis, and 2D rendering — runs on a GPU. Where that GPU lives is the deployment decision described in Deployment shapes. The browser or kiosk shell only captures audio and plays a WebRTC video stream.