Taking a mixture apart

Taking a mix apart, and what “apart” means

Source separation takes a finished mixture and produces estimates of its components: vocals, drums, bass, other. Or, for the case that matters most in video work, speech and everything else.

The standard approach is masking. Convert the audio to a time-frequency representation (a spectrogram) and for each bin, estimate what proportion of the energy there belongs to each source. Multiply the mixture by that mask and you have an estimate of one source. The masks for all sources roughly sum to one, because the energy has to go somewhere.

Spleeter works this way. Demucs is a hybrid: it operates partly on the waveform directly and partly in the spectrogram domain, with a transformer in the middle, which is why it handles transients better than purely spectrogram-based methods.

The word doing the work is “estimate”

Here is the fundamental limit. If a vocal and a guitar both have energy at 800Hz at the same instant, the mixture contains their sum and only their sum. The individual contributions are not recorded anywhere. A separator is guessing the split, from learned statistics about how vocals and guitars usually behave.

Where the sources rarely overlap (a bass line and a hi-hat) separation is close to clean. Where they overlap heavily (a vocal and a string pad in the same register) it is a guess, and the guess shows.

The artefacts, and how to recognise them

Bleed. Traces of one source in another’s stem. Most audible on the drums, because their transients are broadband and hard to assign.

Watery or phasey texture on separated vocals. This is masking error changing from frame to frame: the estimate is slightly different each analysis window, so the residual moves about. It is the same family of problem as musical noise in denoising.

Transient smearing. Percussive attacks lose their edge, because a transient is spread across many frequency bins and the mask cannot be sharp everywhere.

Reverb tails going to the wrong place. The tail of a vocal’s reverb is spectrally similar to the accompaniment and frequently ends up with it, which makes a separated vocal sound drier and more clinical than it was.

The quality metric usually quoted is SDR (signal-to-distortion ratio) in decibels, measured against ground truth on a standard test set. Demucs variants report figures in the region of 9 to 10dB SDR for vocals on the MUSDB18 benchmark. That is good and it is also an average over test material; on your specific mix it may be considerably better or worse.

What it is useful for

• Rescuing dialogue from a music bed on archive material where the stems are lost.
• Removing a cough, a phone or a door from a take that is otherwise good, by separating, editing the offending band, and recombining.
• Repurposing. Pulling a clean vocal for a lyric video, or an instrumental for a backing track.
• Practice and transcription. Isolating a part to learn it.
• Dialogue isolation in a noisy environment, where speech-specific models now outperform general separators.

What it is not good for

Anything where the result will be scrutinised at high quality and at full level. A separated vocal placed nakedly in a quiet mix reveals every artefact; the same vocal sitting inside a new arrangement hides most of them. This is the practical rule: separation output survives being buried and does not survive being exposed.

Recombining without phase problems

A useful property of mask-based separation: the stems sum back to something very close to the original, because the masks were constructed to partition the mixture. So you can separate, process one stem, and add the others back, and the parts you did not touch are essentially unchanged.

This is the technique behind “remove the cough”: separate, fix the stem containing it, sum. It is far less destructive than processing the whole mixture.

The free path

• Demucs: open source, state of the art for music separation, runs on CPU slowly and on a GPU quickly.
• Spleeter: older, faster, lower quality, still useful for a rough split.
• Ultimate Vocal Remover: a free graphical front end bundling several models, which is the easiest starting point.
• DeepFilterNet and RNNoise: speech enhancement rather than music separation, free, and better for the dialogue case.
• ffmpeg has no learned separator, but afftdn and the phase-cancellation trick for centre-channel removal are crude free fallbacks.

The centre-channel trick is worth knowing as a baseline: subtracting one stereo channel from the other cancels anything panned dead centre, which on many older recordings is the lead vocal. It also cancels the bass and kick, and leaves a mono result. It costs nothing and demonstrates how much better the learned methods are.

The limitation

Separation cannot recover information that was destroyed in the sum. Two sources that produced identical energy in the same bin are not separable even in principle; the output is a plausible reconstruction, and where the model is unsure it invents something reasonable rather than reporting uncertainty.

The rights position deserves stating plainly as well. Separating a commercial recording produces a derivative of a copyrighted work, and the fact that a tool makes it easy says nothing about whether the use is permitted. Private study, transcription and practice sit differently from distribution and monetisation, the rules vary by country, and none of it is settled by the existence of the tool. Check what applies where you are before publishing anything derived this way.