Music from a prompt

Sound is a sequence too

The history module told you the transformer did not care what its sequence was made of. Music was one of the first things after text to prove it. Two designs are in use, and both are things you have already met.

The first is the chatbot’s trick applied to sound. A separate small network (a codec) compresses audio into a stream of discrete tokens, a few dozen per second. A transformer is trained to predict the next audio token given the previous ones and a text description. Generation is the next-word game with a different alphabet; the codec turns the tokens back into sound at the end. Meta’s 2023 open model works this way, and so, by every account, do the commercial services.

The second is the image trick. A spectrogram (a picture of sound with time along one axis and pitch along the other) is just an image, and in December 2022 two hobbyists fine-tuned Stable Diffusion on spectrograms and got music out of a picture generator. It was a proof that the mechanism did not care.

What it does well

Asked for “a slow bhangra-influenced pop song about leaving Ludhiana, female vocal, two minutes”, a 2025 service produces exactly that in under a minute: intelligible lyrics, a chorus, a bridge, mixed and mastered to the loudness of a streaming release. The quality is the point; there is no marketing exaggeration in saying a listener would not pick it out of a playlist. Two American services launched in 2023 and 2024 and gained millions of users on that basis.

The reason it is good at this is the reason the chatbot is good at prose. Recorded music in every genre is abundant, and the format (verse, chorus, verse, chorus, bridge, chorus) is one of the most regular structures humans produce.

What it cannot do, and the mechanism

Hold a motif. Ask for a melody that returns transformed in the third verse, and it does not. At a few dozen tokens a second, a three-minute song is many thousands of steps, and the model’s attention over that distance is thin. The chorus that comes back is a similar chorus: the likeliest continuation of “chorus again”, not a recall of the first one. This is the context-window limit from the chatbot module, in sound.

Follow a chart. Give it the chords of a song and ask it to play those, in that order, in that key. It produces something in the right style with chords that are mostly not those. It never learned chords as objects; it learned what recordings tagged with those words sound like.

Edit a bar. Change the fourth bar and keep everything else. There is no fourth bar to point at, for the reason there is no finger to count in the drawing lessons. Services offer extend, remix and replace-a-section tools, and each re-generates a region from scratch and hopes it fits.

Sing clearly at speed. Fast or dense lyrics smear, because the codec’s tokens carry timbre and pitch better than consonants.

The copyright fight

In June 2024 the three largest record companies sued both leading services, alleging that the models had been trained on their catalogues without licence, and asking for damages of up to a hundred and fifty thousand dollars per work. The services’ position was that training on recordings to learn the patterns of music is a fair use, and that their output is new. By late 2025 the first settlements had been announced, turning some of the defendants into licensees while other claims continued.

That is the shape of the argument and this course does not resolve it, because courts and legislatures have not. Two things are not in dispute. Style is not protected by copyright in any major jurisdiction, so a song “in the style of” a living artist is not, by itself, infringement. And recordings are protected, so whether training on them without permission is lawful is exactly the question. A later module puts this case beside the ones about text and images.

The free path

The two commercial services are subscription products. Meta’s 2023 model is open-weights under a non-commercial licence and runs on a modest graphics card; a 2024 open model from the company behind Stable Diffusion generates sound effects and short music under a permissive licence; and a Chinese open model from early 2025 produces full songs with vocals. All run in a free hosted notebook. For everything after generation (cutting, layering, mastering) the free tools are the same ones a studio would use for the job: Audacity for editing, LMMS or Ardour as a full digital audio workstation, against the paid Ableton and Logic that job advertisements name. Every one of the failures above is fixed the same way a human producer fixes them: by editing the audio, not by asking the model again.