One model, many senses

Two ways to build it

The chatbots of 2025 accept a photograph, a voice message, a PDF and a video, and reply in text or speech. The word for this is multimodal, and it covers two very different constructions.

Stitched. Separate models, joined at the text. Speech goes through a transcriber and becomes words; the words go to the language model; its reply goes to a text-to-speech engine. Vision goes through an image encoder whose output is glued into the language model’s context. This is how every voice assistant before 2024 worked, and how most still do.

Native. One model trained from the start on tokens from text, images and audio together, so that a sound and a sentence and a picture live in the same space and the model attends across all of them. The model can produce audio tokens directly as well as text ones.

You can tell which one you are talking to by two things: how long it takes to answer, and whether it knows how you said it.

Why the difference is audible

In the stitched design the transcriber throws away everything but the words. Your tone, your hesitation, the laugh in your voice, the second person in the room: gone before the model sees anything. The reply comes back in a stock voice with stock intonation, and the round trip through three models takes two to five seconds. Conversation at that lag is turn-taking, not talking.

The native design keeps the audio. A major lab’s model of May 2024 responded in about a third of a second on average, close to human conversational timing, and could hear tone, respond to an interruption, laugh, and be asked to speak faster or more warmly, because those were properties of the audio tokens it was predicting rather than instructions to a separate engine. The same lab’s own safety report on that model noted that during testing it had occasionally, unprompted, continued speaking in an imitation of the user’s voice. That is what it means to have a model that predicts audio: a user’s voice is just another likely continuation.

What “watching a video” means

A model advertised as understanding video has not watched it. Video is sampled, typically one frame per second, and each frame is cut into patches and tokenised exactly as in the vision lesson; the audio track is tokenised separately at a few dozen tokens per second. A one-hour video becomes on the order of a million tokens, which is why the labs that offer long video also offer the very long context windows.

So the model saw three thousand six hundred stills and a soundtrack. Anything that happens between frames (a fast gesture, a dropped object, a card trick) is invisible to it. Ask what someone said and it is reading the audio; ask what they did with their hands and it is guessing from stills. Knowing this, you can predict which questions about a video it will answer well before asking them.

What one model buys, and costs

The native design can do things the stitched one cannot: explain a diagram while you point at it with your voice, hear that you sound unsure and slow down, produce a picture and then talk about the picture it produced. It is also more expensive to train, because the audio and image data have to be assembled at the scale of the text data, and it inherits every limitation of every sense at once (the counting blindness of the vision lesson, the cloning capacity of the voice lesson) inside one system, with fewer seams at which to put a check.

The stitched design is cheaper, more controllable and more inspectable. If something goes wrong, you can look at the transcript and see what the model was given. For most business uses that is the right trade, and it is the design behind nearly every customer-service voice system in production.

Free and local

Open native models arrived in 2025 from Chinese and American labs, including one small enough to run on a phone with both camera and microphone. The stitched design is trivially free: an open transcription model from 2022 turns speech into text in dozens of languages on a laptop, any open language model answers, and a lightweight open speech engine reads the reply aloud. It will be slower, and it will not hear your tone. The seams are the price and the benefit.