Music and Audio

011. Sound as numbers

Digital audio is a waveform sampled 44,100 times a second, twice over for stereo. Part 1 put a three-minute song at about 16 million numbers. Unlike an image, those numbers mean almost nothing individually. What we hear as pitch, rhythm and timbre lives in patterns across thousands of samples, from the fifty or so that make one cycle of a bass note to the millions that make a verse.

So every audio model begins with a neural codec: an autoencoder, as in part 1, that turns each second of sound into somewhere between 20 and 75 frames, each a short list of numbers, and can turn them back into sound. The compression is around a hundredfold. What is lost first is what part 1 predicted: the finest detail, which in audio means the crisp top end of cymbals, breath and consonants.

The codec is also where the two designs part company.

• Keep the frames as continuous numbers, and you have a latent that a diffusion model can add noise to and remove it from, exactly as in part 1.
• Snap each frame to the nearest entries in a fixed list of a few thousand, and the song becomes a sequence of tokens that a language model can write.

What this explains

• Fidelity has a ceiling set by the codec. A model cannot produce sound its decoder cannot express. A thin, watery or slightly metallic quality, most audible on vocals and cymbals, is the sound of a codec, and it is the audio counterpart of the washed-out decoder in part 1.
• Length is cheap compared with video. Three minutes of music is a few thousand to ten thousand frames, less than five seconds of video. Whole songs are practical.
• The output is one mixed recording. Drums, bass, vocals and everything else were generated as a single signal. Separate tracks, which any producer would want, have to be pulled apart afterwards by a different model, imperfectly. Section 4 returns to this.

022. Two designs, head to head

The token design

The song is a sequence of codec tokens, and a transformer predicts the next one, as part 1 of the language models module describes. Lyrics and a style description go in the same context as text. The first widely used open music model, from 2023, worked this way and produced about 30 seconds of instrumental music. An open model from early 2025 scaled the idea up to complete songs of several minutes with sung vocals, using one language model for the song’s structure and a second to fill in acoustic detail.

Audio suits this design in a way images do not. It has a natural order, which is time. And music is built from structure over long spans: a chorus that returns, a build that pays off a minute later. Holding a plan over a long sequence is what language models are good at.

The best-known commercial song services do not publish how they work. They are widely reported to be built this way, on the evidence of their makers’ earlier open releases and public statements. Treat that as informed inference and not as fact.

The diffusion design

The whole song is a grid of continuous latent frames, which starts as noise and is denoised together over a few dozen steps, steered by the text. Open models of this kind include one from 2024 for sound effects and short musical pieces of up to 47 seconds, and one from 2025 that generates complete songs with vocals.

That second model’s published figures show what the design buys: up to four minutes of music in about 20 seconds on a data-centre GPU, which its makers put at fifteen times faster than language-model baselines, and it runs on a consumer card with 8 GB of memory. Its makers are also candid about the trade. In their words, the language-model approach excels at aligning lyrics but is slow and prone to structural artefacts, and diffusion is fast but often lacks coherence over long spans.

The trade

Hybrids

As part 1 predicted, the two are being combined along the line the table suggests. A language model sketches the song, its sections, melody and where the words fall, as a coarse token sequence, and a diffusion model renders that sketch into full-quality sound. Research systems from 2025 onwards interleave the two, and it is reasonable to suppose commercial services do something similar. When a service is strong on structure and lyrics and also offers clean editing of a section, this is a likely reason.

033. Songs, sounds and speech

A fourth is arriving from another direction. Part 3 described video models that generate sound with the picture, which for many everyday uses removes the need for a separate sound-effects step.

Speech is the most mature of the three and the least like the rest of this module. Modern speech models are largely token-based and conversational, and they sit closer to the language models module than to this one. Their place here is the risk they carry, which section 6 covers: a usable copy of a person’s voice can be made from under a minute of recording.

044. Working with a song generator

Two prompts, not one

Almost every song generator takes a style prompt and a lyrics field separately, and they do different jobs. The style prompt is a brief: genre, era, tempo, mood, instrumentation, the kind of voice. The lyrics are performed. Structure is given by tags in the lyrics, by convention in square brackets: intro, verse, chorus, bridge, outro. Models trained on tagged lyrics follow them well, and the tags are the main control over a song’s shape.

Left without lyrics, most services write their own with a language model, with the homogenising effect part 7 describes for prompts written by language models.

The editing tools

Each has a counterpart in part 6, because the mechanisms are the same.

One open diffusion model documents all of the first five, plus LoRA training, which shows that they are properties of the design and not of any one product.

Stems

Producers work with stems: separate recordings of drums, bass, vocals and the rest, which can be balanced, processed and replaced individually. A generated song is a single mixed signal. Services that offer stems obtain them by running a separation model over the finished mix. The results are good enough for rebalancing a vocal or making a karaoke version, and usually not clean enough for professional remixing, with traces of each part audible in the others.

This is the audio form of part 1’s observation that the output is flat, and it is the main thing that keeps generated music out of professional production workflows. Models that generate stems directly, or that add a part to an existing recording, exist in research and in some products, and are the development to watch.

055. What goes wrong

• No music theory. A model has no notion of key, chord or bar, only of how recordings sound. Ask for 120 beats a minute in D minor and you will often get roughly that, by association with the words and not by construction. Anything that must synchronise with other music or with picture has to be checked and usually adjusted afterwards.
• Long-range structure. A chorus that returns subtly different, a bridge that never arrives, an ending that simply stops. Diffusion models are weaker here and token models stronger, as section 2 said.
• Vocals. Mispronounced and slurred words, syllables crammed into too few notes, invented words where the lyric was unusual. Names and brand names are the worst case. The vocal is also where codec artefacts are most audible.
• The average. Part 1’s rule about defaults is very audible in music. An unspecific prompt gives glossy, mid-tempo, heavily produced pop with a clean vocal, and lyrics written by a language model rhyme “fire” with “desire”. Distinctive results need distinctive briefs.
• Counting and arrangement. “A string quartet” may contain a piano. Instruments drift in and out in ways no arranger would choose.
• Resemblance. Melodies are short, and the space of pleasant ones is smaller than the space of images. A generated tune that is close to a famous one is more likely than a generated picture that is close to a famous photograph, and music rights holders are far more active. Section 6 returns to this.

As with video, review by listening for one thing at a time: the words, the structure, the ending, and whether it reminds you of something.

066. Rights are different here

The labels and the services

Images were scraped from millions of owners who could not easily act together. Recorded music belongs largely to three major groups with long experience of litigation, and the difference shows.

In June 2024 the major labels sued the two best-known song generators in the United States for training on their recordings. The services argued fair use. In late 2025 the position changed. One major settled with one service at the end of October, and another major settled with the other service in November. Both settlements were coupled with licensing agreements under which the services are to launch new models trained on licensed music, with artists able to opt in, and both brought restrictions on downloading what is generated. Other claims, including those of the largest labels against one of the services, were still being litigated in 2026, and no United States court had ruled on the fair use question.

Alongside this, a number of services train only on music they have licensed or commissioned, and sell that as the point.

What it means for you

• Read the plan, not the homepage. Whether you may use a generated song commercially usually depends on the subscription tier at the moment of generation. Free tiers commonly forbid it. Terms have changed repeatedly as the settlements took effect, so check them when you generate, and keep a record.
• Ownership is as weak as for images. Part 6 explains that in the United States, output from a prompt alone has no copyright protection. Lyrics you wrote yourself are yours. A melody the model produced probably is not.
• Check for resemblance. For anything public, have someone with a good ear listen for a familiar tune, and run the track through a recognition service. Distribution platforms run their own checks and will reject or demonetise matches.
• Distribution platforms have their own rules. Streaming services and music distributors increasingly require AI-generated music to be labelled, and some restrict or exclude it.
• The safest route, where it matters, is a service trained on licensed music that offers an indemnity, or stock and commissioned music as before.

Voices

A speech model can produce a convincing copy of a voice from under a minute of recording. Song models can imitate a singer’s style, and some will imitate a specific voice.

This is where generative media does the most direct harm. Cloned voices are used for fraud, in calls impersonating family members and executives. They are used to put words in the mouths of public figures. For performers, a voice is their livelihood. A voice is also biometric personal data, and a recognisable voice is protected by rights of publicity in many places even where copyright does not reach.

The rule from part 6 applies without softening. No cloning of a real person’s voice without their written consent for the specific use, including colleagues, including “just for the internal demo”. If your organisation uses voice for verification anywhere, such as telephone banking or approving payments, treat that control as broken and replace it.

077. The model families

As of September 2026:

The pattern is the one part 3 found for video. Open models trail the best hosted ones by a clear margin on overall quality, particularly vocals, and offer what hosted ones cannot: they run privately, can be fine-tuned, and do not change under you.

The whiteboard version

Sound is a waveform, squashed by a codec into a few dozen frames a second. Keep those frames as continuous numbers and you can generate a whole song by removing noise, quickly, with easy editing of any section. Snap them to tokens and a language model can write the song from start to finish, slowly, with better structure and lyrics. Products are one, the other or both. Whatever made it, a generated song is a single mixed recording with no separate parts, it has no real notion of key or tempo, and it tends towards the average. The rights position is unlike any other medium: the labels sued, some have licensed, and what you may do depends on the service and the plan. Never copy a real voice without consent.