{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"one-model-many-senses","title":"One model, many senses","url":"https://addaly.com/learn/ai-actually-explained/one-model-many-senses","position":61,"of":92,"minutes":9,"answer":"A \"multimodal\" model is either several models glued together at the text or one model trained on tokens from every sense, and the difference decides whether it hears your tone.","keyIdea":"A \"multimodal\" model is either several models glued together at the text or one model trained on tokens from every sense, and the difference decides whether it hears your tone, how long it takes to reply, and what \"watching a video\" actually means."},"body":{"format":"markdown","text":"## Two ways to build it\n\nThe chatbots of 2025 accept a photograph, a voice message, a PDF and a video, and reply in text or speech. The word for this is multimodal, and it covers two very different constructions.\n\n**Stitched.** Separate models, joined at the text. Speech goes through a transcriber and becomes words; the words go to the language model; its reply goes to a text-to-speech engine. Vision goes through an image encoder whose output is glued into the language model's context. This is how every voice assistant before 2024 worked, and how most still do.\n\n**Native.** One model trained from the start on tokens from text, images and audio together, so that a sound and a sentence and a picture live in the same space and the model attends across all of them. The model can produce audio tokens directly as well as text ones.\n\nYou can tell which one you are talking to by two things: how long it takes to answer, and whether it knows how you said it.\n\n## Why the difference is audible\n\nIn the stitched design the transcriber throws away everything but the words. Your tone, your hesitation, the laugh in your voice, the second person in the room: gone before the model sees anything. The reply comes back in a stock voice with stock intonation, and the round trip through three models takes two to five seconds. Conversation at that lag is turn-taking, not talking.\n\nThe native design keeps the audio. A major lab's model of May 2024 responded in about a third of a second on average, close to human conversational timing, and could hear tone, respond to an interruption, laugh, and be asked to speak faster or more warmly, because those were properties of the audio tokens it was predicting rather than instructions to a separate engine. The same lab's own safety report on that model noted that during testing it had occasionally, unprompted, continued speaking in an imitation of the *user's* voice. That is what it means to have a model that predicts audio: a user's voice is just another likely continuation.\n\n## What \"watching a video\" means\n\nA model advertised as understanding video has not watched it. Video is sampled, typically one frame per second, and each frame is cut into patches and tokenised exactly as in the vision lesson; the audio track is tokenised separately at a few dozen tokens per second. A one-hour video becomes on the order of a million tokens, which is why the labs that offer long video also offer the very long context windows.\n\nSo the model saw three thousand six hundred stills and a soundtrack. Anything that happens between frames (a fast gesture, a dropped object, a card trick) is invisible to it. Ask what someone said and it is reading the audio; ask what they did with their hands and it is guessing from stills. Knowing this, you can predict which questions about a video it will answer well before asking them.\n\n## What one model buys, and costs\n\nThe native design can do things the stitched one cannot: explain a diagram while you point at it with your voice, hear that you sound unsure and slow down, produce a picture and then talk about the picture it produced. It is also more expensive to train, because the audio and image data have to be assembled at the scale of the text data, and it inherits every limitation of every sense at once (the counting blindness of the vision lesson, the cloning capacity of the voice lesson) inside one system, with fewer seams at which to put a check.\n\nThe stitched design is cheaper, more controllable and more inspectable. If something goes wrong, you can look at the transcript and see what the model was given. For most business uses that is the right trade, and it is the design behind nearly every customer-service voice system in production.\n\n## Free and local\n\nOpen native models arrived in 2025 from Chinese and American labs, including one small enough to run on a phone with both camera and microphone. The stitched design is trivially free: an open transcription model from 2022 turns speech into text in dozens of languages on a laptop, any open language model answers, and a lightweight open speech engine reads the reply aloud. It will be slower, and it will not hear your tone. The seams are the price and the benefit."},"check":{"question":"You ask a model to watch a ten-minute video and tell you what the magician did with the coin during a two-second sleight of hand. Why is it likely to be wrong?","options":[{"text":"The video was too long for the model's context window.","correct":false,"tempting":"Misplaces the limit; ten minutes at one frame a second is well within a modern context."},{"text":"The model saw one still per second, so a two-second move is at most two stills and the sleight fell between them.","correct":true},{"text":"The model cannot process video at all and only reads the audio track, so it is guessing from what the magician said.","correct":false,"tempting":"Denies the frames; the model does see frames, but sampled far more sparsely than the action."},{"text":"The model's safety training prevents it from explaining tricks.","correct":false,"tempting":"Invents a refusal; the failure is perceptual, and the model will happily guess."}]},"prev":{"slug":"how-a-watermark-is-hidden","title":"How a watermark is hidden inside the output","url":"https://addaly.com/learn/ai-actually-explained/how-a-watermark-is-hidden"},"next":{"slug":"the-shape-of-a-good-task","title":"The shape of a task it is good at","url":"https://addaly.com/learn/ai-actually-explained/the-shape-of-a-good-task"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'One model, many senses', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/one-model-many-senses (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"One model, many senses","courseTitle":"AI, Actually Explained","module":{"index":6,"slug":"beyond-text","title":"Beyond text: pictures, voices and video"},"answer":null,"keyIdea":"A “multimodal” model is either several models glued together at the text or one model trained on tokens from every sense, and the difference decides whether it hears your tone, how long it takes to reply, and what “watching a video” actually means.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-ways-to-build-it","runs":[{"text":"Two ways to build it"}]},{"t":"p","runs":[{"text":"The chatbots of 2025 accept a photograph, a voice message, a PDF and a video, and reply in text or speech. The word for this is multimodal, and it covers two very different constructions."}]},{"t":"p","runs":[{"text":"Stitched.","b":true},{"text":" Separate models, joined at the text. Speech goes through a transcriber and becomes words; the words go to the language model; its reply goes to a text-to-speech engine. Vision goes through an image encoder whose output is glued into the language model’s context. This is how every voice assistant before 2024 worked, and how most still do."}]},{"t":"p","runs":[{"text":"Native.","b":true},{"text":" One model trained from the start on tokens from text, images and audio together, so that a sound and a sentence and a picture live in the same space and the model attends across all of them. The model can produce audio tokens directly as well as text ones."}]},{"t":"p","runs":[{"text":"You can tell which one you are talking to by two things: how long it takes to answer, and whether it knows how you said it."}]},{"t":"h2","id":"why-the-difference-is-audible","runs":[{"text":"Why the difference is audible"}]},{"t":"p","runs":[{"text":"In the stitched design the transcriber throws away everything but the words. Your tone, your hesitation, the laugh in your voice, the second person in the room: gone before the model sees anything. The reply comes back in a stock voice with stock intonation, and the round trip through three models takes two to five seconds. Conversation at that lag is turn-taking, not talking."}]},{"t":"p","runs":[{"text":"The native design keeps the audio. A major lab’s model of May 2024 responded in about a third of a second on average, close to human conversational timing, and could hear tone, respond to an interruption, laugh, and be asked to speak faster or more warmly, because those were properties of the audio tokens it was predicting rather than instructions to a separate engine. The same lab’s own safety report on that model noted that during testing it had occasionally, unprompted, continued speaking in an imitation of the "},{"text":"user’s","i":true},{"text":" voice. That is what it means to have a model that predicts audio: a user’s voice is just another likely continuation."}]},{"t":"h2","id":"what-watching-a-video-means","runs":[{"text":"What “watching a video” means"}]},{"t":"p","runs":[{"text":"A model advertised as understanding video has not watched it. Video is sampled, typically one frame per second, and each frame is cut into patches and tokenised exactly as in the vision lesson; the audio track is tokenised separately at a few dozen tokens per second. A one-hour video becomes on the order of a million tokens, which is why the labs that offer long video also offer the very long context windows."}]},{"t":"p","runs":[{"text":"So the model saw three thousand six hundred stills and a soundtrack. Anything that happens between frames (a fast gesture, a dropped object, a card trick) is invisible to it. Ask what someone said and it is reading the audio; ask what they did with their hands and it is guessing from stills. Knowing this, you can predict which questions about a video it will answer well before asking them."}]},{"t":"h2","id":"what-one-model-buys-and-costs","runs":[{"text":"What one model buys, and costs"}]},{"t":"p","runs":[{"text":"The native design can do things the stitched one cannot: explain a diagram while you point at it with your voice, hear that you sound unsure and slow down, produce a picture and then talk about the picture it produced. It is also more expensive to train, because the audio and image data have to be assembled at the scale of the text data, and it inherits every limitation of every sense at once (the counting blindness of the vision lesson, the cloning capacity of the voice lesson) inside one system, with fewer seams at which to put a check."}]},{"t":"p","runs":[{"text":"The stitched design is cheaper, more controllable and more inspectable. If something goes wrong, you can look at the transcript and see what the model was given. For most business uses that is the right trade, and it is the design behind nearly every customer-service voice system in production."}]},{"t":"h2","id":"free-and-local","runs":[{"text":"Free and local"}]},{"t":"p","runs":[{"text":"Open native models arrived in 2025 from Chinese and American labs, including one small enough to run on a phone with both camera and microphone. The stitched design is trivially free: an open transcription model from 2022 turns speech into text in dozens of languages on a laptop, any open language model answers, and a lightweight open speech engine reads the reply aloud. It will be slower, and it will not hear your tone. The seams are the price and the benefit."}]}],"check":{"question":[{"text":"You ask a model to watch a ten-minute video and tell you what the magician did with the coin during a two-second sleight of hand. Why is it likely to be wrong?"}],"options":[{"text":[{"text":"The video was too long for the model’s context window."}],"correct":false,"tempting":[{"text":"Misplaces the limit; ten minutes at one frame a second is well within a modern context."}]},{"text":[{"text":"The model saw one still per second, so a two-second move is at most two stills and the sleight fell between them."}],"correct":true,"tempting":null},{"text":[{"text":"The model cannot process video at all and only reads the audio track, so it is guessing from what the magician said."}],"correct":false,"tempting":[{"text":"Denies the frames; the model does see frames, but sampled far more sparsely than the action."}]},{"text":[{"text":"The model’s safety training prevents it from explaining tricks."}],"correct":false,"tempting":[{"text":"Invents a refusal; the failure is perceptual, and the model will happily guess."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"how-a-watermark-is-hidden","title":"How a watermark is hidden inside the output","minutes":9},"next":{"slug":"the-shape-of-a-good-task","title":"The shape of a task it is good at","minutes":9}}}