Speech: ASR, Diarization, TTS, Speech-to-Speech

Speech models connect sound and text. Automatic speech recognition (ASR) turns audio into text, text-to-speech (TTS) turns text into audio, and voice assistants chain both around a language model. These models are built much like LLMs, so many LLM serving tools apply, but the way their speed is judged is different. For live use, what counts is how fast the first piece arrives and how many conversations one GPU can hold at once, not raw tokens per second.

This lesson covers ASR (transcription), diarization (who spoke when), TTS (speech synthesis), and speech-to-speech models. Afterwards you can pick a deployment shape for live transcription versus hour-long files, compute a real-time factor, size a TTS deployment by concurrent streams, and explain why most voice agents are pipelines of three models.

Terms from earlier lessons. TTFT is time to first token. TensorRT-LLM is NVIDIA's optimized inference engine (Inference Engines). MIG (Multi-Instance GPU) cuts one large GPU into smaller isolated slices (Instances, Multi-GPU, MIG, and Other Accelerators). In-flight batching adds and removes requests from a running batch at each step, instead of waiting for a whole batch to finish. Autoregressive means producing one token at a time, each depending on the earlier ones.

Example 1: Whisper and the encoder-decoder design

ASR models take audio and produce text, powering transcription and dictation apps. The most popular open ASR model is Whisper, released by OpenAI. It supports dozens of languages with accurate transcription. Whisper comes in several sizes, but even the largest and best one has only 1.55B parameters. It runs very fast on a fraction of a big GPU such as an H100 using MIG slices. Many other sizes, variants, distillations, and quantizations exist, but in practice the highest-quality models, Whisper 3 Large and Whisper 3 Turbo, meet most latency budgets.

Whisper is an encoder-decoder model:

• Encoder: takes processed audio, a log-Mel spectrogram (a picture of how the energy at each pitch changes over time), and converts it to audio features.
• Decoder: takes the audio features and produces text tokens.

graph LR
    A["Audio waveform"] --> S["log-Mel spectrogram"]
    S --> E["Encoder: audio features"]
    E --> D["Decoder: text tokens, one at a time"]
    D --> T["Transcript"]

Nearly all inference time goes to the decoder, an autoregressive transformer much like an LLM. That is helpful, because the tools for optimizing LLM decode already exist. The main one is TensorRT-LLM, which gives in-flight batching for the decoder and an optimized C++ runtime with efficient CUDA kernels. It works especially well on recent architectures, Hopper and Blackwell (GPU Generations), which makes MIG slices an even better fit for ASR.

Example 2: real-time transcription of a single chunk

One use of Whisper is live transcription in a dictation app or a voice agent. For live Whisper, measure the round-trip time for one chunk of audio to be transcribed. A good target is 200 milliseconds, the average human reaction time.

With Whisper on an optimized TensorRT-LLM engine, there is little left to do at the runtime level. Most of the remaining gains come from orchestration and infrastructure. The biggest product improvement is streaming, which is implemented in the API server layer, not in the model runtime. The client opens a WebSocket (a long-lived two-way connection, see Client Code: Latency, Async Inference, Streaming) and sends audio continuously while text comes back continuously, so transcription happens live instead of after a recording ends.

The ASR runtime itself does not change. A Voice Activity Detection (VAD) model watches the incoming stream and cuts it into separate chunks for the ASR model. Inference runs on each chunk as usual and the text is streamed back over the WebSocket.

graph LR
    M["Microphone stream"] -->|WebSocket| V["VAD: cut into chunks"]
    V --> W["Whisper on one GPU"]
    W -->|text over WebSocket| U["Live text"]

This setup serves several concurrent streams and keeps transcription in order. When every chunk of a stream runs on the same GPU, the output of the previous chunk can be used as the prefix for the next, which improves transcription quality.

Example 3: long files and the real-time factor

Whisper handles only 30-second chunks, so long files such as hour-long podcasts need a different design. Speed on long files is measured by the real-time factor (RTF), a confusing name because bigger is better: RTF = audio length / processing time. A fast typist who transcribes an hour of audio in 30 minutes has RTF = 60 / 30 = 2X. A Whisper deployment tuned for long files can transcribe an hour in under four seconds:

RTF = audio duration / processing time
    = 3,600 s / 3.6 s
    = 1,000X          (an hour of audio in about 3.6 seconds)

This takes a multi-step pipeline:

1. A VAD model, on its own dedicated hardware, removes silence and cuts the audio into meaningful segments. Cutting by fixed time intervals would risk splitting words in half.
2. The chunks are processed in parallel, ideally on several GPUs (or several MIG slices). RTF improves roughly linearly with the number of GPUs. Each GPU processes many chunks at once with in-flight batching for high utilization.
3. The chunk transcripts are stitched back together by timestamp.

graph LR
    F["Long audio file"] --> V["VAD: remove silence, cut segments"]
    V --> G1["GPU 1: chunks"]
    V --> G2["GPU 2: chunks"]
    V --> G3["GPU 3: chunks"]
    G1 --> S["Stitch by timestamp"]
    G2 --> S
    G3 --> S

A worked case: one hour is 3,600 / 30 = 120 chunks of 30 seconds at most (VAD segments are usually shorter, so the real count differs). On 8 GPUs, each gets about 120 / 8 = 15 chunks, and if one GPU needs t seconds for 15 chunks the whole hour takes about t seconds. On 16 GPUs it would take roughly t / 2, which is the linear improvement.

Running chunks in parallel removes the option of using the previous chunk's text as a prefix. Other quality techniques more than make up for it. ASR output can be checked for hallucinations such as repeated words or phrases by measuring the compression ratio (repeated text compresses well) and the words per minute of the output. When a chunk looks wrong:

1. Re-run it with a higher temperature. This is counterintuitive, since higher temperatures usually cause more hallucination, but the goal is to break a cycle of repeated words and get a different output.
2. Re-chunk the whole audio, or a segment of it, into smaller chunks and transcribe again.

In practice these techniques remove the need to pass the previous text as a prefix, which allows efficient and accurate parallel transcription of long files.

Example 4: diarization

Diarization labels a transcript with who is speaking when. It is a neighbor of transcription. Diarization models sort audio by voice characteristics, then segment and cluster across the file to mark the timestamps where the speaker changes.

These models are a different class from Whisper. Whisper is an encoder-decoder transformer. Diarization systems such as pyannote audio are pipelines of classic ML models, with models for segmentation, embedding, and clustering. Speeding up diarization means making each model fast and orchestrating the pipeline well.

graph LR
    A["Audio"] --> S["Segmentation"]
    S --> E["Embedding of each segment"]
    E --> C["Clustering: same voice, same speaker"]
    C --> L["Speaker labels with timestamps"]

Because it is an ML pipeline, standard tools such as PyTorch and pyannote apply, along with optimizations such as Torch compilation. Even highly optimized diarization takes at least twice as long as transcribing the same file.

Example 5: text-to-speech (TTS)

Text-to-speech models, also called speech synthesis models, take text and produce audio, specifically speech. In 2025, open models such as Orpheus TTS brought very lifelike speech to open models. Many companies fine-tuned Orpheus for better vocal quality and product-specific voices, which led to wide adoption of open models in voice AI.

Modern TTS models are fine-tuned LLMs. Orpheus TTS comes from Llama 3.2 3B, so many LLM runtime optimizations carry over. TTS models are small. Orpheus at three billion parameters is on the larger end, so like ASR they run efficiently on MIG slices of H100 GPUs. Unlike ASR models, which usually run in FP16, TTS weights and KV cache can be quantized to FP8 for extra speed (Quantization), on top of the kernels and in-flight batching of TensorRT-LLM.

An LLM is trained to speak this way: its vocabulary is expanded by tens of thousands of encoded audio tokens, and it is trained on pairs of text input and tokenized audio output. So a TTS deployment also needs an audio decoder that turns output audio tokens into a waveform (such as the SNAC decoder used with Orpheus).

graph LR
    T["Text"] --> L["Fine-tuned LLM: text to audio tokens"]
    L --> D["Audio decoder (for example SNAC)"]
    D --> W["Waveform"]

The audio decoder can become a bottleneck. It should be implemented in PyTorch and compiled for the target GPU, and it should use dynamic batching with a short timeout (for example 15 milliseconds). In-flight batching is not possible for the audio decoder.

TTS has its own metrics:

As with TTFT, the goal is to minimize TTFB. For Orpheus, as low as 150 milliseconds on one H100 is possible.

TPS has a different goal here. The generated tokens become an audio waveform, and depending on the model it takes about 80 to 100 tokens per second to generate audio in real time. Producing tokens faster than that gives no benefit. Instead, optimizations raise throughput as the number of concurrent real-time streams a GPU can sustain. The more users one GPU supports, the lower the cost per user. A worked case with illustrative numbers:

real time needs   : 100 tokens per second per stream
GPU capacity      : 2,400 tokens per second in total (illustrative)
concurrent streams: 2,400 / 100 = 24 real-time streams on one GPU
cost per stream   : one GPU price / 24

If a tuning change lifts the GPU to 3,600 tokens per second, the same GPU serves 36 streams, and the cost per stream falls by one third.

Example 6: streaming TTS, long inputs, and speech-to-speech

Most TTS jobs are real time. As with ASR, performance for real-time systems comes less from the runtime (already optimized with TensorRT-LLM, quantization, and a compiled SNAC decoder) and more from infrastructure. Streaming over WebSockets is again the biggest gain compared with sending text and receiving audio in separate pieces. After testing how many concurrent real-time streams the engine can produce, set the batch size and the number of active WebSocket connections to the same number. That keeps usage high but stable.

TTS is rarely used outside real time. For a batch job, such as converting a large set of documents to audio for accessibility, TTS models do not handle long inputs well: speech starts to degrade after about 30 seconds.

Speech-to-speech models take audio in and generate audio out. Today most voice systems use a cascading design: an ASR model, an LLM, and a TTS model work in a pipeline to listen, think, and respond. These pipelines add helper components such as VAD and embedding models to make conversation natural and to add context.

graph LR
    M["User speech"] --> V["VAD"]
    V --> A["ASR"]
    A --> L["LLM"]
    L --> T["TTS"]
    T --> S["Spoken reply"]
    E["Embedding model: context"] --> L

Speech-to-speech models, such as OpenAI's gpt-realtime, extend a core LLM with audio input and output, which merges the pipeline into one model. This is possible because ASR, LLM, and TTS models have such similar architectures, especially in the decoder. At the time of writing (2026) there are no commercially viable open speech-to-speech models. Closed options such as gpt-realtime are significantly less capable and more expensive than cascading multi-model setups. Research is active, and this new modality will soon need its own kind of inference engineering.

Interview points

• Whisper is an encoder-decoder model with at most 1.55B parameters. Almost all time goes to the autoregressive decoder, so TensorRT-LLM (in-flight batching, C++ runtime, CUDA kernels) is the main tool, and MIG slices on H100, Hopper, or Blackwell fit well.
• Live transcription targets about 200 ms per chunk. Streaming is an API-server feature: WebSocket in and out, VAD to chunk the stream, same GPU per stream so the previous output can be the next prefix.
• Long files: RTF is audio length divided by processing time (an hour in under four seconds is about 1,000X). The pipeline is VAD on its own hardware, parallel chunks over GPUs or MIGs with in-flight batching (RTF scales roughly linearly with GPUs), and stitching by timestamp. Whisper takes 30-second chunks.
• Detect hallucinations through compression ratio and words per minute. Fix by re-running with a higher temperature or by re-chunking.
• Diarization is a pipeline of classic models (segmentation, embedding, clustering) such as pyannote audio, taking at least twice as long as transcription even when optimized.
• TTS models are fine-tuned LLMs (Orpheus from Llama 3.2 3B) with audio tokens added to the vocabulary and a separate audio decoder (SNAC). Weights and KV cache can use FP8. The decoder is compiled PyTorch with dynamic batching with about a 15 ms timeout, and no in-flight batching.
• TTS metrics: TTFB (about 150 ms for Orpheus on one H100), time to first sentence, TPS. Real time needs only about 80 to 100 tokens per second per stream, so scale by concurrent streams. Speech degrades past about 30 seconds of input.
• Cascading voice agents (ASR, LLM, TTS, plus VAD and embeddings) beat closed speech-to-speech models such as gpt-realtime in capability and cost today.

Summary

• ASR (Whisper), diarization, TTS (Orpheus), and speech-to-speech all reuse LLM-like architectures and tools, but each needs its own latency measures.
• Whisper is encoder-decoder, small, and decoder-bound, and runs on TensorRT-LLM and MIG slices.
• Live ASR is a streaming problem: WebSocket, VAD chunks, in-order processing on one GPU. Long-file ASR is a parallel pipeline measured by RTF.
• Diarization is a classic ML pipeline and is slower than transcription.
• TTS needs an audio decoder, FP8 quantization, TTFB and time-to-first-sentence metrics, and scaling by concurrent real-time streams.
• Voice agents are cascades of ASR, LLM, and TTS, while single-model speech-to-speech is still immature.