{"course":{"slug":"ai-video","title":"Video with AI","url":"https://addaly.com/learn/ai-video","level":"intermediate","track":"create"},"lesson":{"slug":"models-that-generate-their-own-audio","title":"When the model makes the sound too","url":"https://addaly.com/learn/ai-video/models-that-generate-their-own-audio","position":48,"of":66,"minutes":8,"answer":"Native audio generation produces synchronised ambience and dialogue in one pass, which is remarkable and usually unusable as a final track, because you cannot separate, replace or adjust anything in it afterwards.","keyIdea":"Native audio generation produces synchronised ambience and dialogue in one pass, which is remarkable and usually unusable as a final track, because you cannot separate, replace or adjust anything in it afterwards."},"body":{"format":"markdown","text":"## What is now possible\n\nSeveral flagship models generate a soundtrack along with the picture: ambience matched to the scene, footsteps roughly in sync, and in some cases spoken dialogue with lip movement that matches. Google's Veo line is the most prominent example of the capability.\n\nThis is a genuine technical achievement and it is useful. It is also, for most production work, a draft rather than a deliverable, and the reason is not quality.\n\n## The reason is that it is baked\n\nThe audio arrives as a single mixed track welded to the picture. That means:\n\n- **You cannot adjust the balance.** If the ambience is too loud under the dialogue, it is too loud permanently.\n- **You cannot replace one element.** A wrong footstep sound cannot be swapped without replacing everything.\n- **You cannot extend it across a cut.** Each clip has its own audio, which stops at the clip boundary, which is exactly the digital silence problem from two lessons ago, arriving at every edit point.\n- **You cannot match it to the next shot.** Two generated clips have two unrelated ambiences. Cutting between them produces an audible jump at every cut, which is more noticeable than a visual mismatch.\n- **You cannot localise it.** A dialogue line in one language is not translatable without regenerating the shot.\n\nA mixed track is the end of an audio process, not the middle of one. Receiving it at the start of your edit means you have inherited somebody else's final mix for a film that does not exist yet.\n\n## Where it is the right answer\n\nBe specific rather than dismissive, because there are real cases:\n\n- **Single-clip social content.** One shot, no cuts, posted as is. No continuity problem exists, so the objections above do not apply.\n- **Previsualisation and pitching.** A moving storyboard with sound communicates far better than one without, and nobody is grading it.\n- **A reference track.** Generate with audio, then use that audio as a guide for what to build properly: it tells you what the model thought the scene sounded like, which is sometimes a good idea you would not have had.\n- **Speed over control.** A deadline where a finished-enough clip now beats a better clip tomorrow.\n\n## The trade-off with the first frame\n\nThe awkward part, and it is worth knowing before you plan a shoot.\n\nOn several models the native audio path is tied to text prompting rather than to a supplied first frame. If you want the model's audio, you may have to give up image-to-video, which means giving up the still, the composite, the repaired hand, the character sheet and most of what this course has built.\n\nThat is a large price for a soundtrack you will probably replace. Decide per shot, and be clear that it is a trade rather than a free addition.\n\n## Extracting and rebuilding\n\nIf you do generate with audio and want to work properly afterwards:\n\n```bash\n# separate the audio from a generated clip\nffmpeg -i shot.mp4 -vn -acodec pcm_s16le shot_audio.wav\n# strip audio from the picture for a clean edit\nffmpeg -i shot.mp4 -an -c:v copy shot_silent.mp4\n```\n\nThen treat the extracted audio as source material: pull out the one ambience you liked, loop it as a bed, and build the rest in the normal way. Source separation tools (Demucs and similar, free and open) will split a mixed track into rough stems, which sometimes rescues a good element from a bad mix.\n\n## Where this is going, without predicting it\n\nThe obvious improvement would be generated audio delivered as separate stems: dialogue, ambience, effects, each on its own track. That is what a production actually needs, and it is a reasonable thing to hope for rather than a thing to plan around.\n\nUntil it exists, the working assumption stands: generate the picture, build the sound. It is more work, it is entirely free, and it is the half of the job where a beginner can match a professional result with a phone and an afternoon.\n\n---\n\nToday: generate one clip with native audio, extract the audio with FFmpeg, and see whether any single element in it is worth keeping as a bed."},"check":{"question":"Why is a model's natively generated soundtrack usually a draft rather than a deliverable for a multi-shot piece?","options":[{"text":"Generated audio is produced at a lower sample rate than the picture's frame rate requires, so it drifts out of sync over a long timeline.","correct":false,"tempting":"Invents a sync-drift mechanism; the audio is generated against the same clip and stays locked to it."},{"text":"Native audio is generated from the text prompt rather than from the picture, so it describes the scene the prompt asked for rather than the scene that was produced.","correct":false,"tempting":"Overstates the disconnection; the audio is generated jointly with the picture and is usually well matched to it within a clip."},{"text":"Licensing terms for generated audio differ from those for generated video on most platforms, so the soundtrack cannot be used commercially even where the picture can.","correct":false,"tempting":"Asserts a terms-of-service distinction that is not generally how these products are licensed."},{"text":"It arrives as one mixed track per clip, so nothing can be rebalanced or replaced and each cut produces an audible jump between unrelated ambiences.","correct":true}]},"prev":{"slug":"sound-design-fixes-bad-physics","title":"The viewer believes the sound","url":"https://addaly.com/learn/ai-video/sound-design-fixes-bad-physics"},"next":{"slug":"the-edit-is-the-work","title":"The generator makes footage; the editor makes the video","url":"https://addaly.com/learn/ai-video/the-edit-is-the-work"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the model makes the sound too', in Video with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-video/models-that-generate-their-own-audio (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the model makes the sound too","courseTitle":"Video with AI","module":{"index":5,"slug":"sound-and-speech","title":"The half of video that is not picture"},"answer":null,"keyIdea":"Native audio generation produces synchronised ambience and dialogue in one pass, which is remarkable and usually unusable as a final track, because you cannot separate, replace or adjust anything in it afterwards.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-is-now-possible","runs":[{"text":"What is now possible"}]},{"t":"p","runs":[{"text":"Several flagship models generate a soundtrack along with the picture: ambience matched to the scene, footsteps roughly in sync, and in some cases spoken dialogue with lip movement that matches. Google’s Veo line is the most prominent example of the capability."}]},{"t":"p","runs":[{"text":"This is a genuine technical achievement and it is useful. It is also, for most production work, a draft rather than a deliverable, and the reason is not quality."}]},{"t":"h2","id":"the-reason-is-that-it-is-baked","runs":[{"text":"The reason is that it is baked"}]},{"t":"p","runs":[{"text":"The audio arrives as a single mixed track welded to the picture. That means:"}]},{"t":"ul","items":[[{"text":"You cannot adjust the balance.","b":true},{"text":" If the ambience is too loud under the dialogue, it is too loud permanently."}],[{"text":"You cannot replace one element.","b":true},{"text":" A wrong footstep sound cannot be swapped without replacing everything."}],[{"text":"You cannot extend it across a cut.","b":true},{"text":" Each clip has its own audio, which stops at the clip boundary, which is exactly the digital silence problem from two lessons ago, arriving at every edit point."}],[{"text":"You cannot match it to the next shot.","b":true},{"text":" Two generated clips have two unrelated ambiences. Cutting between them produces an audible jump at every cut, which is more noticeable than a visual mismatch."}],[{"text":"You cannot localise it.","b":true},{"text":" A dialogue line in one language is not translatable without regenerating the shot."}]]},{"t":"p","runs":[{"text":"A mixed track is the end of an audio process, not the middle of one. Receiving it at the start of your edit means you have inherited somebody else’s final mix for a film that does not exist yet."}]},{"t":"h2","id":"where-it-is-the-right-answer","runs":[{"text":"Where it is the right answer"}]},{"t":"p","runs":[{"text":"Be specific rather than dismissive, because there are real cases:"}]},{"t":"ul","items":[[{"text":"Single-clip social content.","b":true},{"text":" One shot, no cuts, posted as is. No continuity problem exists, so the objections above do not apply."}],[{"text":"Previsualisation and pitching.","b":true},{"text":" A moving storyboard with sound communicates far better than one without, and nobody is grading it."}],[{"text":"A reference track.","b":true},{"text":" Generate with audio, then use that audio as a guide for what to build properly: it tells you what the model thought the scene sounded like, which is sometimes a good idea you would not have had."}],[{"text":"Speed over control.","b":true},{"text":" A deadline where a finished-enough clip now beats a better clip tomorrow."}]]},{"t":"h2","id":"the-trade-off-with-the-first-frame","runs":[{"text":"The trade-off with the first frame"}]},{"t":"p","runs":[{"text":"The awkward part, and it is worth knowing before you plan a shoot."}]},{"t":"p","runs":[{"text":"On several models the native audio path is tied to text prompting rather than to a supplied first frame. If you want the model’s audio, you may have to give up image-to-video, which means giving up the still, the composite, the repaired hand, the character sheet and most of what this course has built."}]},{"t":"p","runs":[{"text":"That is a large price for a soundtrack you will probably replace. Decide per shot, and be clear that it is a trade rather than a free addition."}]},{"t":"h2","id":"extracting-and-rebuilding","runs":[{"text":"Extracting and rebuilding"}]},{"t":"p","runs":[{"text":"If you do generate with audio and want to work properly afterwards:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# separate the audio from a generated clip\nffmpeg -i shot.mp4 -vn -acodec pcm_s16le shot_audio.wav\n# strip audio from the picture for a clean edit\nffmpeg -i shot.mp4 -an -c:v copy shot_silent.mp4"},{"t":"p","runs":[{"text":"Then treat the extracted audio as source material: pull out the one ambience you liked, loop it as a bed, and build the rest in the normal way. Source separation tools (Demucs and similar, free and open) will split a mixed track into rough stems, which sometimes rescues a good element from a bad mix."}]},{"t":"h2","id":"where-this-is-going-without-predicting-it","runs":[{"text":"Where this is going, without predicting it"}]},{"t":"p","runs":[{"text":"The obvious improvement would be generated audio delivered as separate stems: dialogue, ambience, effects, each on its own track. That is what a production actually needs, and it is a reasonable thing to hope for rather than a thing to plan around."}]},{"t":"p","runs":[{"text":"Until it exists, the working assumption stands: generate the picture, build the sound. It is more work, it is entirely free, and it is the half of the job where a beginner can match a professional result with a phone and an afternoon."}]},{"t":"rule"},{"t":"p","runs":[{"text":"Today: generate one clip with native audio, extract the audio with FFmpeg, and see whether any single element in it is worth keeping as a bed."}]}],"check":{"question":[{"text":"Why is a model’s natively generated soundtrack usually a draft rather than a deliverable for a multi-shot piece?"}],"options":[{"text":[{"text":"Generated audio is produced at a lower sample rate than the picture’s frame rate requires, so it drifts out of sync over a long timeline."}],"correct":false,"tempting":[{"text":"Invents a sync-drift mechanism; the audio is generated against the same clip and stays locked to it."}]},{"text":[{"text":"Native audio is generated from the text prompt rather than from the picture, so it describes the scene the prompt asked for rather than the scene that was produced."}],"correct":false,"tempting":[{"text":"Overstates the disconnection; the audio is generated jointly with the picture and is usually well matched to it within a clip."}]},{"text":[{"text":"Licensing terms for generated audio differ from those for generated video on most platforms, so the soundtrack cannot be used commercially even where the picture can."}],"correct":false,"tempting":[{"text":"Asserts a terms-of-service distinction that is not generally how these products are licensed."}]},{"text":[{"text":"It arrives as one mixed track per clip, so nothing can be rebalanced or replaced and each cut produces an audible jump between unrelated ambiences."}],"correct":true,"tempting":null}],"order":[2,0,1,3],"why":null},"prev":{"slug":"sound-design-fixes-bad-physics","title":"The viewer believes the sound","minutes":8},"next":{"slug":"the-edit-is-the-work","title":"The generator makes footage; the editor makes the video","minutes":10}}}