FROOT extension · V1

Voice & Speech AI

Design the conversation around the latency budget.

Real-time conversational systems on Azure: streaming STT, LLM orchestration, TTS, telephony, barge-in, safety, and the evidence required to keep first audio near 800ms.

800ms

Design target

Measured stages

The human constraint

A three-second answer feels broken

Text RAG can take several seconds without destroying the interaction. Voice cannot. Silence is part of the product, so every architecture decision must preserve a human-perceptible response budget.

Turn detection

200–400ms

STT finalization

50–150ms

LLM first token

200–500ms

TTS first audio

100–300ms

First sound reaches the user~850ms

Canonical Azure stack

Listen → reason → speak → observe

Azure AI Speech

Streaming STT, neural TTS, and batch transcription.

Communication Services

PSTN connectivity, Call Automation, SIP, and Direct Routing.

Azure OpenAI

Fast routing with a smaller model and richer response generation where needed.

Container Apps or AKS

Pre-warmed compute that can absorb call spikes without cold-start pauses.

Application Insights

Per-call timing, errors, abandonment, and component-level latency.

Streaming STT · Python

Authenticate without embedding a key

Use Managed Identity in production, stream partial results, shorten end-of-speech detection deliberately, and refresh the token before it expires.

speech.py · continuous recognition

import azure.cognitiveservices.speech as speechsdk
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
token = credential.get_token("https://cognitiveservices.azure.com/.default")
speech_config = speechsdk.SpeechConfig(
    auth_token=f"aad#{RESOURCE_ID}#{token.token}",
    region="eastus2",
)
speech_config.speech_recognition_language = "en-US"
speech_config.set_property(
    speechsdk.PropertyId.SpeechServiceConnection_EndSilenceTimeoutMs,
    "500",
)

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioConfig(use_default_microphone=True),
)
recognizer.recognizing.connect(lambda event: print("[partial]", event.result.text))
recognizer.recognized.connect(lambda event: print("[final]", event.result.text))
recognizer.start_continuous_recognition()

Patterns that work

Start TTS at the first sentence boundary while the next sentence is still generating.
Use short filler phrases only when they honestly mask a known processing step.
Filter backchannels such as “uh-huh” and “okay” before treating them as a new turn.
Keep conversational memory to the last 5–10 turns unless retrieval supplies older context.
Use a smaller model for routing and a stronger model only for content that needs it.
Use two-region active-active design when voice availability is business critical.

Anti-patterns to avoid

Waiting for the complete LLM response before beginning TTS.
Using the largest model for every routing decision.
Sending system prompts larger than 2,000 tokens on every turn.
Allowing audio queues to grow without a hard bound or cancellation policy.
Storing API keys in source or shipping them to the browser.
Crossing regions between STT, LLM, and TTS without measuring every hop.
Retaining raw transcripts without PII redaction and retention controls.

Evidence and safety

Measure the call, not only the model

Latency evidence

Capture turn detection, STT final, first token, first audio, and end-to-end percentiles per call.

Conversation quality

Track interruption recovery, abandonment, fallback frequency, transcript confidence, and task completion.

Safety boundary

Redact PII, apply retention controls, evaluate accent bias, and filter unsafe generated audio before playback.

Continue the path

Play 04