{"course":{"slug":"motion-and-audio","title":"Motion, Sound and Music","url":"https://addaly.com/learn/motion-and-audio","level":"intermediate","track":"create"},"lesson":{"slug":"separating-a-mix","title":"Taking a mixture apart","url":"https://addaly.com/learn/motion-and-audio/separating-a-mix","position":68,"of":72,"minutes":10,"answer":"Separation estimates a mask per source per time-frequency bin, so where two sources had energy in the same bin the split is a learned guess rather than a recovery.","keyIdea":"Separation estimates a mask per source per time-frequency bin, so where two sources had energy in the same bin the split is a learned guess rather than a recovery; and because the masks partition the mixture, the stems sum back cleanly, which is what makes separate-fix-recombine a gentler repair than processing the whole mix."},"body":{"format":"markdown","text":"## Taking a mix apart, and what \"apart\" means\n\nSource separation takes a finished mixture and produces estimates of its components: vocals, drums, bass, other. Or, for the case that matters most in video work, speech and everything else.\n\nThe standard approach is **masking**. Convert the audio to a time-frequency representation (a spectrogram) and for each bin, estimate what proportion of the energy there belongs to each source. Multiply the mixture by that mask and you have an estimate of one source. The masks for all sources roughly sum to one, because the energy has to go somewhere.\n\nSpleeter works this way. **Demucs** is a hybrid: it operates partly on the waveform directly and partly in the spectrogram domain, with a transformer in the middle, which is why it handles transients better than purely spectrogram-based methods.\n\n## The word doing the work is \"estimate\"\n\nHere is the fundamental limit. If a vocal and a guitar both have energy at 800Hz at the same instant, the mixture contains their sum and only their sum. The individual contributions are not recorded anywhere. A separator is *guessing* the split, from learned statistics about how vocals and guitars usually behave.\n\nWhere the sources rarely overlap (a bass line and a hi-hat) separation is close to clean. Where they overlap heavily (a vocal and a string pad in the same register) it is a guess, and the guess shows.\n\n## The artefacts, and how to recognise them\n\n**Bleed.** Traces of one source in another's stem. Most audible on the drums, because their transients are broadband and hard to assign.\n\n**Watery or phasey texture** on separated vocals. This is masking error changing from frame to frame: the estimate is slightly different each analysis window, so the residual moves about. It is the same family of problem as musical noise in denoising.\n\n**Transient smearing.** Percussive attacks lose their edge, because a transient is spread across many frequency bins and the mask cannot be sharp everywhere.\n\n**Reverb tails going to the wrong place.** The tail of a vocal's reverb is spectrally similar to the accompaniment and frequently ends up with it, which makes a separated vocal sound drier and more clinical than it was.\n\nThe quality metric usually quoted is **SDR** (signal-to-distortion ratio) in decibels, measured against ground truth on a standard test set. Demucs variants report figures in the region of 9 to 10dB SDR for vocals on the MUSDB18 benchmark. That is good and it is also an average over test material; on your specific mix it may be considerably better or worse.\n\n## What it is useful for\n\n- **Rescuing dialogue from a music bed** on archive material where the stems are lost.\n- **Removing a cough, a phone or a door** from a take that is otherwise good, by separating, editing the offending band, and recombining.\n- **Repurposing.** Pulling a clean vocal for a lyric video, or an instrumental for a backing track.\n- **Practice and transcription.** Isolating a part to learn it.\n- **Dialogue isolation in a noisy environment**, where speech-specific models now outperform general separators.\n\n## What it is not good for\n\nAnything where the result will be scrutinised at high quality and at full level. A separated vocal placed nakedly in a quiet mix reveals every artefact; the same vocal sitting inside a new arrangement hides most of them. This is the practical rule: separation output survives being buried and does not survive being exposed.\n\n## Recombining without phase problems\n\nA useful property of mask-based separation: the stems sum back to something very close to the original, because the masks were constructed to partition the mixture. So you can separate, process one stem, and add the others back, and the parts you did not touch are essentially unchanged.\n\nThis is the technique behind \"remove the cough\": separate, fix the stem containing it, sum. It is far less destructive than processing the whole mixture.\n\n## The free path\n\n- **Demucs**: open source, state of the art for music separation, runs on CPU slowly and on a GPU quickly.\n- **Spleeter**: older, faster, lower quality, still useful for a rough split.\n- **Ultimate Vocal Remover**: a free graphical front end bundling several models, which is the easiest starting point.\n- **DeepFilterNet** and **RNNoise**: speech enhancement rather than music separation, free, and better for the dialogue case.\n- **ffmpeg** has no learned separator, but `afftdn` and the phase-cancellation trick for centre-channel removal are crude free fallbacks.\n\nThe centre-channel trick is worth knowing as a baseline: subtracting one stereo channel from the other cancels anything panned dead centre, which on many older recordings is the lead vocal. It also cancels the bass and kick, and leaves a mono result. It costs nothing and demonstrates how much better the learned methods are.\n\n## The limitation\n\nSeparation cannot recover information that was destroyed in the sum. Two sources that produced identical energy in the same bin are not separable even in principle; the output is a plausible reconstruction, and where the model is unsure it invents something reasonable rather than reporting uncertainty.\n\nThe rights position deserves stating plainly as well. Separating a commercial recording produces a derivative of a copyrighted work, and the fact that a tool makes it easy says nothing about whether the use is permitted. Private study, transcription and practice sit differently from distribution and monetisation, the rules vary by country, and none of it is settled by the existence of the tool. Check what applies where you are before publishing anything derived this way."},"check":{"question":"A separated vocal sounds acceptable inside a new arrangement and unpleasantly watery when exposed on its own in a quiet passage. What is the mechanism behind the watery quality?","options":[{"text":"The separator works in the time domain, so it introduces phase rotation in the vocal that is masked by other instruments and revealed when they are removed.","correct":false,"tempting":"Hybrid separators do operate partly on the waveform, but the artefact described comes from frame-to-frame mask variation rather than phase rotation."},{"text":"The mask estimate changes slightly from frame to frame, so the residual error moves about: the same family of problem as musical noise in denoising.","correct":true},{"text":"The vocal's reverb tail has been assigned to the accompaniment stem, and the missing tail is what makes the exposed vocal sound unnatural.","correct":false,"tempting":"A real and common separation error, but it produces dryness rather than a moving, watery texture."},{"text":"The separator's output is reconstructed from a compressed latent representation rather than from the original spectrogram, so it is a resynthesis of the vocal rather than an extraction of it, and every resynthesis carries codec artefacts that only become audible without masking content around them.","correct":false,"tempting":"Mask-based separators multiply the original mixture rather than resynthesising from a latent, which is why the stems sum back to the source."}]},"prev":{"slug":"captions-and-word-error-rate","title":"Five per cent, and which five per cent","url":"https://addaly.com/learn/motion-and-audio/captions-and-word-error-rate"},"next":{"slug":"generated-music-and-unsettled-law","title":"Generated music, and three separate legal questions","url":"https://addaly.com/learn/motion-and-audio/generated-music-and-unsettled-law"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Taking a mixture apart', in Motion, Sound and Music. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/motion-and-audio/separating-a-mix (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Taking a mixture apart","courseTitle":"Motion, Sound and Music","module":{"index":8,"slug":"machines-that-make-sound-and-motion","title":"Machines that make sound and motion"},"answer":null,"keyIdea":"Separation estimates a mask per source per time-frequency bin, so where two sources had energy in the same bin the split is a learned guess rather than a recovery; and because the masks partition the mixture, the stems sum back cleanly, which is what makes separate-fix-recombine a gentler repair than processing the whole mix.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"taking-a-mix-apart-and-what-apart-means","runs":[{"text":"Taking a mix apart, and what “apart” means"}]},{"t":"p","runs":[{"text":"Source separation takes a finished mixture and produces estimates of its components: vocals, drums, bass, other. Or, for the case that matters most in video work, speech and everything else."}]},{"t":"p","runs":[{"text":"The standard approach is "},{"text":"masking","b":true},{"text":". Convert the audio to a time-frequency representation (a spectrogram) and for each bin, estimate what proportion of the energy there belongs to each source. Multiply the mixture by that mask and you have an estimate of one source. The masks for all sources roughly sum to one, because the energy has to go somewhere."}]},{"t":"p","runs":[{"text":"Spleeter works this way. "},{"text":"Demucs","b":true},{"text":" is a hybrid: it operates partly on the waveform directly and partly in the spectrogram domain, with a transformer in the middle, which is why it handles transients better than purely spectrogram-based methods."}]},{"t":"h2","id":"the-word-doing-the-work-is-estimate","runs":[{"text":"The word doing the work is “estimate”"}]},{"t":"p","runs":[{"text":"Here is the fundamental limit. If a vocal and a guitar both have energy at 800Hz at the same instant, the mixture contains their sum and only their sum. The individual contributions are not recorded anywhere. A separator is "},{"text":"guessing","i":true},{"text":" the split, from learned statistics about how vocals and guitars usually behave."}]},{"t":"p","runs":[{"text":"Where the sources rarely overlap (a bass line and a hi-hat) separation is close to clean. Where they overlap heavily (a vocal and a string pad in the same register) it is a guess, and the guess shows."}]},{"t":"h2","id":"the-artefacts-and-how-to-recognise-them","runs":[{"text":"The artefacts, and how to recognise them"}]},{"t":"p","runs":[{"text":"Bleed.","b":true},{"text":" Traces of one source in another’s stem. Most audible on the drums, because their transients are broadband and hard to assign."}]},{"t":"p","runs":[{"text":"Watery or phasey texture","b":true},{"text":" on separated vocals. This is masking error changing from frame to frame: the estimate is slightly different each analysis window, so the residual moves about. It is the same family of problem as musical noise in denoising."}]},{"t":"p","runs":[{"text":"Transient smearing.","b":true},{"text":" Percussive attacks lose their edge, because a transient is spread across many frequency bins and the mask cannot be sharp everywhere."}]},{"t":"p","runs":[{"text":"Reverb tails going to the wrong place.","b":true},{"text":" The tail of a vocal’s reverb is spectrally similar to the accompaniment and frequently ends up with it, which makes a separated vocal sound drier and more clinical than it was."}]},{"t":"p","runs":[{"text":"The quality metric usually quoted is "},{"text":"SDR","b":true},{"text":" (signal-to-distortion ratio) in decibels, measured against ground truth on a standard test set. Demucs variants report figures in the region of 9 to 10dB SDR for vocals on the MUSDB18 benchmark. That is good and it is also an average over test material; on your specific mix it may be considerably better or worse."}]},{"t":"h2","id":"what-it-is-useful-for","runs":[{"text":"What it is useful for"}]},{"t":"ul","items":[[{"text":"Rescuing dialogue from a music bed","b":true},{"text":" on archive material where the stems are lost."}],[{"text":"Removing a cough, a phone or a door","b":true},{"text":" from a take that is otherwise good, by separating, editing the offending band, and recombining."}],[{"text":"Repurposing.","b":true},{"text":" Pulling a clean vocal for a lyric video, or an instrumental for a backing track."}],[{"text":"Practice and transcription.","b":true},{"text":" Isolating a part to learn it."}],[{"text":"Dialogue isolation in a noisy environment","b":true},{"text":", where speech-specific models now outperform general separators."}]]},{"t":"h2","id":"what-it-is-not-good-for","runs":[{"text":"What it is not good for"}]},{"t":"p","runs":[{"text":"Anything where the result will be scrutinised at high quality and at full level. A separated vocal placed nakedly in a quiet mix reveals every artefact; the same vocal sitting inside a new arrangement hides most of them. This is the practical rule: separation output survives being buried and does not survive being exposed."}]},{"t":"h2","id":"recombining-without-phase-problems","runs":[{"text":"Recombining without phase problems"}]},{"t":"p","runs":[{"text":"A useful property of mask-based separation: the stems sum back to something very close to the original, because the masks were constructed to partition the mixture. So you can separate, process one stem, and add the others back, and the parts you did not touch are essentially unchanged."}]},{"t":"p","runs":[{"text":"This is the technique behind “remove the cough”: separate, fix the stem containing it, sum. It is far less destructive than processing the whole mixture."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"ul","items":[[{"text":"Demucs","b":true},{"text":": open source, state of the art for music separation, runs on CPU slowly and on a GPU quickly."}],[{"text":"Spleeter","b":true},{"text":": older, faster, lower quality, still useful for a rough split."}],[{"text":"Ultimate Vocal Remover","b":true},{"text":": a free graphical front end bundling several models, which is the easiest starting point."}],[{"text":"DeepFilterNet","b":true},{"text":" and "},{"text":"RNNoise","b":true},{"text":": speech enhancement rather than music separation, free, and better for the dialogue case."}],[{"text":"ffmpeg","b":true},{"text":" has no learned separator, but "},{"text":"afftdn","code":true},{"text":" and the phase-cancellation trick for centre-channel removal are crude free fallbacks."}]]},{"t":"p","runs":[{"text":"The centre-channel trick is worth knowing as a baseline: subtracting one stereo channel from the other cancels anything panned dead centre, which on many older recordings is the lead vocal. It also cancels the bass and kick, and leaves a mono result. It costs nothing and demonstrates how much better the learned methods are."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"Separation cannot recover information that was destroyed in the sum. Two sources that produced identical energy in the same bin are not separable even in principle; the output is a plausible reconstruction, and where the model is unsure it invents something reasonable rather than reporting uncertainty."}]},{"t":"p","runs":[{"text":"The rights position deserves stating plainly as well. Separating a commercial recording produces a derivative of a copyrighted work, and the fact that a tool makes it easy says nothing about whether the use is permitted. Private study, transcription and practice sit differently from distribution and monetisation, the rules vary by country, and none of it is settled by the existence of the tool. Check what applies where you are before publishing anything derived this way."}]}],"check":{"question":[{"text":"A separated vocal sounds acceptable inside a new arrangement and unpleasantly watery when exposed on its own in a quiet passage. What is the mechanism behind the watery quality?"}],"options":[{"text":[{"text":"The separator works in the time domain, so it introduces phase rotation in the vocal that is masked by other instruments and revealed when they are removed."}],"correct":false,"tempting":[{"text":"Hybrid separators do operate partly on the waveform, but the artefact described comes from frame-to-frame mask variation rather than phase rotation."}]},{"text":[{"text":"The mask estimate changes slightly from frame to frame, so the residual error moves about: the same family of problem as musical noise in denoising."}],"correct":true,"tempting":null},{"text":[{"text":"The vocal’s reverb tail has been assigned to the accompaniment stem, and the missing tail is what makes the exposed vocal sound unnatural."}],"correct":false,"tempting":[{"text":"A real and common separation error, but it produces dryness rather than a moving, watery texture."}]},{"text":[{"text":"The separator’s output is reconstructed from a compressed latent representation rather than from the original spectrogram, so it is a resynthesis of the vocal rather than an extraction of it, and every resynthesis carries codec artefacts that only become audible without masking content around them."}],"correct":false,"tempting":[{"text":"Mask-based separators multiply the original mixture rather than resynthesising from a latent, which is why the stems sum back to the source."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"captions-and-word-error-rate","title":"Five per cent, and which five per cent","minutes":11},"next":{"slug":"generated-music-and-unsettled-law","title":"Generated music, and three separate legal questions","minutes":11}}}