GPT-4o: Real-Time Multimodality for Assistants
GPT-4o, announced on 13 May 2024, brings text, vision, and audio into a single end-to-end network. We put the verified numbers in context — 320 ms average voice latency, half the price of GPT-4 Turbo, five times the rate limits — and distinguish clearly between the API available today and the promise of the launch demos.
Why voice assistants feel slow
Human conversation follows a tight cadence: speakers hand over turns within a few hundred milliseconds. Voice assistants have never met that timing budget. ChatGPT's Voice Mode, launched in September 2023, is a pipeline of three models: speech recognition transcribes audio, GPT-3.5 or GPT-4 processes text, and a synthesis model reads the answer aloud. Average response latency is 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. Users notice the delay: a multi-second pause feels like a machine, not a conversation partner.
Latency, however, is only half the problem. The pipeline gives the language model a transcript and nothing else. Tone, multiple speakers, and background noise disappear before the model processes the input; on output, it cannot laugh, sing, or control emphasis. These losses are structural: no amount of tuning can restore information already discarded between three serial model stages.
One model for every modality
On 13 May 2024 OpenAI announced GPT-4o. The "o" stands for omni. It is a single neural network trained end-to-end across text, vision, and audio. All inputs and all outputs pass through the same model. It accepts any combination of text, audio, and image input and can generate text, audio, and image output. There is no transcription step and no synthesis step left to serialize.
On text, GPT-4o matches GPT-4 Turbo in English and code and improves on non-English languages. It scores 88.7% on 0-shot CoT MMLU, a new high. A new tokenizer compresses many languages: OpenAI's German sample drops from 34 to 29 tokens (1.2x), Hindi 2.9x, Gujarati 4.4x. Fewer tokens mean lower cost and faster generation for non-English workloads.
Half the price at twice the speed
The API numbers are concrete. GPT-4o costs $5 per million input tokens and $15 per million output tokens — half the price of GPT-4 Turbo. It generates tokens roughly twice as fast and ships with five times the rate limits, ramping up to 10 million tokens per minute. The context window stays at 128K tokens.
Model IDs gpt-4o and gpt-4o-2024-05-13 are live in the Chat Completions, Assistants, and Batch APIs. For applications already running on GPT-4 Turbo, switching is a one-line change that halves the bill. For German-language workloads the new tokenizer compounds the saving: the same German text bills roughly 15% fewer tokens in OpenAI's sample.
| Property | GPT-4o | GPT-4 Turbo |
|---|---|---|
| Input price per 1M tokens | $5 | $10 |
| Output price per 1M tokens | $15 | $30 |
| Token generation | ~2x faster | baseline |
| Rate limits | up to 10M tokens/min (5x) | baseline |
| Context window | 128K | 128K |
| Knowledge cutoff | October 2023 | December 2023 |
Voice latency at conversational speed
GPT-4o responds to audio input in as little as 232 milliseconds, 320 milliseconds on average. OpenAI compares this to human response times in conversation. Against Voice Mode's 5.4 seconds with GPT-4, that is a reduction of roughly 94%. The gain comes from removing pipeline stages, not from a faster pipeline.
These figures need two caveats. First, they come from OpenAI's product demo and do not describe an API property that can be benchmarked publicly today, because the audio endpoint is not yet available. Second, latency below 300 milliseconds does not make the model smarter; it makes the interaction feel more natural. Both matter, but they should not be conflated.
What real-time multimodality changes
Sub-second turn-taking changes the interaction model, not just the speed. An assistant that answers in 300 milliseconds can be interrupted mid-sentence and must handle barge-in. It can hear tone and hesitation, not just words. The launch demos showed live translation between English and Italian and a model reacting to camera input and a shared screen in real time.
Architecturally, the design point therefore shifts from request-response to a continuous conversation loop. Every layer must stream, state must survive interruptions, and grounding now spans live audio and video alongside retrieved documents. We treat latency as a budget assigned to each processing hop; native multimodality removes two of those hops entirely.
API access versus product features
The demo and the API are not the same thing. As of today, 15 May 2024, developers get GPT-4o as a text and vision model. Audio and video input and output in the API are announced for "a small group of trusted partners" in the coming weeks. No public speech-to-speech endpoint exists yet.
On the product side, GPT-4o is rolling out to ChatGPT's free tier; Plus subscribers get five times the message limits. The new Voice Mode built on GPT-4o enters alpha for Plus users in the coming weeks, alongside a ChatGPT desktop app for macOS. An assistant that needs speech today still needs the old pipeline — GPT-4o merely shortens its middle leg.
What GPT-4o does not change
The context window stays at 128K tokens. The knowledge cutoff is October 2023 — two months earlier than GPT-4 Turbo's December 2023. Text intelligence is on par with GPT-4 Turbo, not beyond it; this release buys speed, price, and modality, not reasoning. Hallucination and prompt-injection risks carry over unchanged, and the full rate limits are still being ramped up.
Native audio also adds new risk surface: voice impersonation and emotional manipulation. OpenAI states that audio output will launch with a selection of preset voices only. Independent evaluations of the audio modality do not exist yet — the published latency and quality figures are the vendor's own.
Outlook from May 2024
Two predictions from where we stand. First, a public speech-to-speech API will arrive within months, and competitors will follow with natively multimodal models; the three-model pipeline becomes a legacy pattern except where the intermediate transcript is required for auditability. Second, GPT-4 cost $30/$60 per million tokens in March 2023; GPT-4o costs $5/$15 fourteen months later. Frontier-level inference keeps commoditizing.
For our work at Blue IT Systems, this leads to a concrete decision: voice assistants still use the pipeline pattern for now, but the middle stage remains replaceable so that a native audio endpoint can be integrated without rebuilding the system. Real-time multimodality is no longer only a research demo; it is a foreseeable deployment question.