When the model makes the sound too

What is now possible

Several flagship models generate a soundtrack along with the picture: ambience matched to the scene, footsteps roughly in sync, and in some cases spoken dialogue with lip movement that matches. Google’s Veo line is the most prominent example of the capability.

This is a genuine technical achievement and it is useful. It is also, for most production work, a draft rather than a deliverable, and the reason is not quality.

The reason is that it is baked

The audio arrives as a single mixed track welded to the picture. That means:

• You cannot adjust the balance. If the ambience is too loud under the dialogue, it is too loud permanently.
• You cannot replace one element. A wrong footstep sound cannot be swapped without replacing everything.
• You cannot extend it across a cut. Each clip has its own audio, which stops at the clip boundary, which is exactly the digital silence problem from two lessons ago, arriving at every edit point.
• You cannot match it to the next shot. Two generated clips have two unrelated ambiences. Cutting between them produces an audible jump at every cut, which is more noticeable than a visual mismatch.
• You cannot localise it. A dialogue line in one language is not translatable without regenerating the shot.

A mixed track is the end of an audio process, not the middle of one. Receiving it at the start of your edit means you have inherited somebody else’s final mix for a film that does not exist yet.

Where it is the right answer

Be specific rather than dismissive, because there are real cases:

• Single-clip social content. One shot, no cuts, posted as is. No continuity problem exists, so the objections above do not apply.
• Previsualisation and pitching. A moving storyboard with sound communicates far better than one without, and nobody is grading it.
• A reference track. Generate with audio, then use that audio as a guide for what to build properly: it tells you what the model thought the scene sounded like, which is sometimes a good idea you would not have had.
• Speed over control. A deadline where a finished-enough clip now beats a better clip tomorrow.

The trade-off with the first frame

The awkward part, and it is worth knowing before you plan a shoot.

On several models the native audio path is tied to text prompting rather than to a supplied first frame. If you want the model’s audio, you may have to give up image-to-video, which means giving up the still, the composite, the repaired hand, the character sheet and most of what this course has built.

That is a large price for a soundtrack you will probably replace. Decide per shot, and be clear that it is a trade rather than a free addition.

Extracting and rebuilding

If you do generate with audio and want to work properly afterwards:

# separate the audio from a generated clip
ffmpeg -i shot.mp4 -vn -acodec pcm_s16le shot_audio.wav
# strip audio from the picture for a clean edit
ffmpeg -i shot.mp4 -an -c:v copy shot_silent.mp4

Then treat the extracted audio as source material: pull out the one ambience you liked, loop it as a bed, and build the rest in the normal way. Source separation tools (Demucs and similar, free and open) will split a mixed track into rough stems, which sometimes rescues a good element from a bad mix.

Where this is going, without predicting it

The obvious improvement would be generated audio delivered as separate stems: dialogue, ambience, effects, each on its own track. That is what a production actually needs, and it is a reasonable thing to hope for rather than a thing to plan around.

Until it exists, the working assumption stands: generate the picture, build the sound. It is more work, it is entirely free, and it is the half of the job where a beginner can match a professional result with a phone and an afternoon.

Today: generate one clip with native audio, extract the audio with FFmpeg, and see whether any single element in it is worth keeping as a bed.