How Generative Media Models Actually Work

011. The whole thing in one sentence

A diffusion model is a function that takes a noisy sample, a noise level and a text prompt, and returns its best guess at the noise in that sample. For an image model the sample is a picture. For the others it is a clip, a stretch of audio or a shape. Generation is that function called in a loop: guess the noise, remove some of it, call again. Everything else in this part is detail inside that sentence.

As code, the entire runtime looks like this:

text = encode_text(prompt)
latent = random_noise(seed)           # a small grid of pure noise
for t in schedule:                    # 20 to 50 steps, from all noise to none
    noise_guess = model(latent, t, text)
    latent = remove_some(latent, noise_guess, t)
image = decode(latent)                # expand the small grid into pixels

Three things are worth noticing before we open up model.

• The model is a pure function. Its weights are fixed at training time, exactly as a language model’s are. Nothing you type changes them, and it remembers nothing between one output and the next.
• The whole thing is worked on at once. There is no left to right and no top to bottom, and in video and music no start to finish. Every step touches every part of the output, which starts vague everywhere and sharpens everywhere together.
• Chance enters once, at the start. The starting noise comes from a random number generator with a seed. With most samplers, the same seed, prompt, settings and model give the same result.

The pipeline has five stages:

The prompt becomes a set of vectors. A seed becomes a small grid of noise. The denoiser, which is the large neural network, is run a few dozen times, and each run leaves the grid a little cleaner. A decoder then expands the grid into pixels, or into frames, sound or geometry. Sections 2 to 6 take these stages in turn. Section 7 describes the other design, and section 8 draws out what both imply.

022. Media as numbers

An image is a grid of pixels, and each pixel is three numbers: how much red, green and blue. A picture 1,024 pixels on each side is 1,024 × 1,024 × 3 numbers, a little over three million of them. The other media are larger, and each has a shape of its own.

Why not predict the next number

A language model works because text comes in a modest number of discrete pieces with a natural order. Part 1 of the language models module builds everything on that. Raw media offers neither.

• There are too many. Three million values, produced one at a time, would mean three million passes through the network for one picture, and a hundred times that for a few seconds of video.
• There is often no order. The top-left pixel is not the “start” of a photograph, and a 3D object has no first triangle. Anything produced early would have to be right before the rest had been decided. Sound and video do have an order, which is time, and section 7 shows that this matters.
• The values are continuous. A token is one of a fixed list. A pixel value or a sound sample is a quantity, and a nearly right quantity is fine, which is a different kind of problem from choosing the right word.

Diffusion sidesteps all three. It treats the whole sample as one object and improves all of it together, over a small number of passes.

What this explains

• Size drives cost. Doubling the width and height of an image means four times as many numbers, and the work grows at least as fast. Every extra second of video or audio adds in the same way. A large output is slow and expensive for the same reason a long prompt is.
• Models have home sizes. A model is trained on samples of particular sizes and shapes: certain resolutions, certain clip lengths. Ask an older image model for a much larger or wider picture than it was trained on and it tends to repeat itself: two heads, a second horizon, a doubled subject. Ask a video model for a clip far longer than its training clips and it drifts or loops. Newer models cope with a wider range, but each still has one.
• The output is flat. The result is the finished thing and nothing more. An image has no layers, no separate objects and no editable text. A song arrives as one mixed recording and not as separate instruments. A 3D object arrives as one fused surface with no moving parts. Anything that looks like structure is only an appearance of it.
• Bigger usually means a second stage. Very large outputs are normally made by generating at the home size and then enlarging with a separate model: an upscaler for images, a frame interpolator for smoother video. Generating directly at the final size is slow and drifts away from what the model was trained on.

033. Learning to remove noise

The training game

Take an image from the training set, with its caption. For other media, take a clip, a recording or a shape, with its description. Pick a noise level at random, from “barely touched” to “nothing but static”. Add that much random noise to the image. Show the model the noisy image, the noise level and the caption, and ask it to predict the noise that was added. Compare its guess with the truth, and adjust every weight slightly to reduce the error.

That is the whole of training, repeated billions of times. As with a language model, the labels are free: the noise was added by the training code, so the right answer is always known. No person marks anything.

The task changes character with the noise level, and one network learns all of it.

• At high noise almost nothing of the picture survives. To guess the noise, the model has to guess what picture could plausibly be underneath, and the caption is most of what it has to go on. This is where it learns composition: what goes where.
• At low noise the picture is nearly all there. The task becomes local: clean up this edge, this patch of skin, this strand of hair. This is where it learns texture and detail.

From training to generating

Generation runs the game backwards. Start with pure noise, which is the highest noise level. Ask the model what noise it sees. Remove part of that guess, which leaves a slightly less noisy grid. Tell the model the new, lower noise level and ask again. After a few dozen rounds no noise is left, and what remains is a picture that was never in the training set.

Why many small steps

It is fair to ask why the model cannot remove all the noise in one go. From pure static, many different pictures are equally plausible for “a dog in a park”. The best single guess that is least wrong on average is the average of all of them, and the average of many sharp pictures is a grey blur.

Small steps avoid this. Each step commits a little: a dark mass here, a horizon there. That commitment narrows what the next step has to consider, in the same way that each token a language model emits narrows what can follow. By the later steps there is only one picture left to be consistent with, and the guesses become sharp.

Coarse first, fine last

Because of the way the task changes with noise level, a picture forms in a fixed order. The first few steps settle the layout and the main masses of light and colour. The middle steps settle shapes and objects. The last steps add texture and fine detail.

The same order holds in every medium. In a video, the early steps settle what is in the shot and how it moves, and the late steps settle surfaces. In music, they settle tempo, structure and the broad arrangement first, and the character of each sound last. In 3D, the overall shape comes before the surface detail.

This has practical uses.

• A preview after a few steps tells you the composition. If the layout is wrong at step 8 of 30, it will still be wrong at step 30. Tools that show a live preview let you abandon bad attempts early.
• Late changes cannot move things. Anything that intervenes in the final steps can only restyle or sharpen what is already there.
• Starting part way is a technique. If you begin from a real image with some noise added, instead of from pure noise, the model keeps the layout of that image and reinvents the detail. This is image-to-image, and part 6 builds on it. Its counterparts are video-to-video and audio-to-audio.

What it is trained on

Image models came first because the data was there. The first generation of open image models learned from several billion images scraped from the public web, each paired with whatever text sat beside it, usually the alt text written for accessibility or search engines. One widely used public collection held about 5.85 billion such pairs.

Alt text is a poor description of a picture. It is short, often wrong, and often a product name or a string of keywords. Models trained on it learned to respond to keywords, because keywords were what they saw.

Later models are trained on captions written by another AI model, which looks at each image and describes it in full sentences: the subject, the setting, the lighting, where things are. One influential 2023 report found that this change alone produced a large improvement in how well a model follows a detailed prompt. It is the main reason newer models can be prompted in ordinary prose, and part 7 returns to it.

The other media have far less described data to learn from, and this shapes each of them. Video with accurate descriptions is scarcer than images. Music is plentiful, and almost all of it is owned by someone, which part 4 shows has consequences. Good 3D models number in the millions at most, against billions of images, and part 5 explains how that field works around it.

044. Working in a compressed space

Running a large network over every pixel, dozens of times per image, is expensive. The 2022 design called latent diffusion removed most of that cost, and almost every current model, in every medium, uses it.

Two models, trained separately

First, train an autoencoder. It has two halves. The encoder squeezes an image into a much smaller grid of numbers. The decoder expands that grid back into an image. The pair is trained on one goal: what comes out should look like what went in. The small grid in the middle is called the latent, and the space of all such grids is the latent space.

In the first widely used open model, an image of 512 × 512 pixels became a latent of 64 × 64 positions with 4 numbers at each. That is 786,432 numbers reduced to 16,384, a factor of 48. Each position in the latent stands for an 8 × 8 block of pixels.

Second, train the diffusion model entirely inside that latent space. It never sees a pixel. Training images are encoded to latents, noise is added to the latents, and the network learns to remove noise from latents. At generation time the loop in section 1 runs on the small grid, and the decoder is called once at the end.

The compression works because most of what is in a photograph is predictable fine texture. The autoencoder learns to store “grass here” and “skin here” compactly and to reinvent the blades and pores on the way out. The diffusion model is left with the part that matters: what the picture is of.

The same trick in other media

Each medium has its own autoencoder, and the saving is larger the more redundant the medium is.

In every case the diffusion model works only on the small version, and a decoder produces the frames, the waveform or the mesh at the end.

This saving is what put image generation on consumer hardware. Stable Diffusion, released with open weights in August 2022, was a latent diffusion model small enough to run on a gaming graphics card, and the ecosystem described in part 6 grew from that.

What this explains

• Small things go wrong first. Each latent position covers a block of pixels, so anything only a few pixels across is below the resolution the model actually thinks in. The decoder has to invent it. A face filling the frame is fine. The same face in the back of a crowd is often mangled. The counterparts elsewhere are small text, fast flickering detail in video, the crisp top end of cymbals and consonants in audio, and thin structures such as wires and railings in 3D.
• The decoder has a look of its own. Colour, contrast and fine sharpness are partly set by the decoder and not by the main model. This is why tools for open models let you swap the decoder, usually labelled VAE, and why the wrong one gives washed-out or oddly tinted images.
• Newer models keep more per position. Later designs store 16 numbers at each latent position in place of 4. The compression is milder, and fine detail, small faces and lettering all improved as a result.
• “Latent” tools work on the miniature. Features described as latent upscaling, latent blending or latent noise operate on the small grid before decoding. They are fast for the same reason the model is.

055. How the prompt steers

The text encoder

The prompt is handled by a text encoder, a separate model that is trained first and then frozen. It turns the prompt into a sequence of vectors, one per token of the prompt. The denoiser never sees your words, only those vectors.

Inside the denoiser, the image consults the text through cross-attention. This is the attention mechanism from part 1 of the language models module, with one change: the queries come from positions in the image, and the keys and values come from the tokens of the prompt. Each region of the picture asks, in effect, “which words are about me?”, and pulls in information from those words. A region that is becoming a dog attends to “dog” and to “golden”. A region that is becoming sky attends to “sunset”.

Which reader a model has

The choice of text encoder decides what kind of prompt a model understands, and models differ widely.

CLIP was the encoder of the first generation. It was trained to tell which caption goes with which image, and for that task a caption is close to a bag of keywords. “A red cube on a blue sphere” and “a blue cube on a red sphere” look almost the same to it. It also has a hard limit of 77 tokens: anything beyond that is cut off or handled by workarounds in the tool.

Later models add or substitute an encoder that understands sentences. Many use two or three encoders together, taking style and subject from one and structure from another.

This is the root of part 7. A list of comma-separated tags suits a model with a CLIP encoder that was trained on keyword captions. Full descriptive sentences suit a model with a language-model encoder that was trained on long written captions. Using the wrong style is not fatal, but it wastes the prompt.

More than text

Nothing in this arrangement requires the steering signal to be words. Anything that can be turned into vectors can be fed to the denoiser the same way: a reference image, a starting frame, a melody, a depth map, a pose. Image-to-video models are steered mainly by their first frame. Music models take lyrics as well as a style description. Image-to-3D models take a picture and no words at all. Part 6 is largely about the non-text signals, and they are usually stronger than the text.

Guidance

A model trained this way follows the prompt only loosely. The fix, used by most models, is classifier-free guidance.

At each step the denoiser is run twice: once with your prompt, and once with an empty prompt. The difference between the two guesses is the direction in which the prompt is pulling. The sampler then moves in that direction, exaggerated by a factor called the guidance scale, often labelled CFG.

• Too low, around 1 to 3 for a first-generation model, and the prompt is a suggestion. Images are varied and natural but may ignore half of what you asked for.
• Moderate, around 5 to 8 for those models, is the usual working range.
• Too high, and the image becomes harsh: oversaturated colour, heavy contrast, a burnt, overcooked look, and less variety between seeds.

The right range differs by model, and newer models generally want lower values than older ones. Treat the figures above as an illustration and use the range the model’s publisher recommends.

Negative prompts

Guidance explains a feature that otherwise seems odd. In the second run, the empty prompt can be replaced by a prompt of your own. The sampler then pushes towards your main prompt and away from that one. This is a negative prompt: a description of what you do not want.

Two things follow. A negative prompt works only in a model that runs guidance this way. Several fast or distilled models skip the second run to halve the cost, and in those the negative prompt box does nothing, even if the tool still shows one. And writing “no cars” in the main prompt does not work in any keyword-style model, because the encoder sees the word “cars” and the model adds them. What you do not want goes in the negative prompt or is left unsaid.

guess = guess_empty + scale * (guess_prompt - guess_empty)

066. Samplers, steps and seeds

The network only ever guesses noise. Everything about how that guess is used lives in ordinary code outside it, called the sampler or scheduler. This is the image counterpart of the sampler in a language model: no learning, just arithmetic, and a great deal of influence on the result.

Steps and samplers

Mathematically, generation means following a path from noise to image, where the network supplies the direction at each point. A sampler is a method for following that path in a limited number of steps. Better methods stay closer to the true path with fewer steps.

Tools for open models offer a long list of samplers with names such as Euler, DDIM, DPM++ and UniPC. The differences that matter in practice are few.

• Most good samplers converge on the same picture. Given the same seed and enough steps, they arrive at nearly the same image. They differ in how few steps they need to get there, typically 20 to 30.
• “Ancestral” samplers do not converge. Those with an “a” or “ancestral” in the name add fresh noise at every step. The picture keeps changing as you add steps, and results are harder to reproduce.
• More steps stop helping. Beyond the point of convergence, extra steps cost time and change nothing useful. Fifty steps is rarely better than thirty.
• Few-step models are a different product. Models labelled Turbo, Lightning or LCM have been distilled to produce an image in one to eight steps. They are several times faster, which makes live previews and interactive tools possible. They usually give up some variety and fine detail, and most ignore negative prompts.

The seed

The starting noise is the only random input, and the seed determines it. Change the seed and you get a different picture for the same prompt. Keep the seed and you get the same one.

The early steps settle composition, and they are driven largely by the starting noise. So the seed mostly decides layout: where the subject stands, which way it faces, where the light falls. The prompt decides what the things are.

That gives two working modes.

• To explore, vary the seed. Generate a batch with different seeds and one prompt. You are sampling different pictures from the same description.
• To refine, fix the seed. Keep the seed and change one thing in the prompt. Now the difference between two images is caused by your change and not by chance. Without this, prompt tuning is guesswork, because you cannot tell a better prompt from a luckier draw.

Reproducing an image exactly needs everything to match: model and version, seed, prompt, negative prompt, sampler, steps, guidance and size. Even then, different hardware or software versions can produce small differences. Tools for open models write these settings into the image file’s metadata, which is worth keeping.

The settings at a glance

Hosted services hide most of these and choose for you. They are still there. When a service offers “variations”, it is changing the seed. When it offers “stylise” or “prompt strength”, it is some form of guidance.

077. The other design: tokens

How it works

Section 2 said that raw media cannot be produced one number at a time. The way round this is to stop working with raw numbers. A codec, which is an autoencoder with a twist, compresses the media as section 4 described and then snaps each piece of the compressed version to the nearest entry in a fixed list of a few thousand. A picture becomes a few thousand tokens. A second of audio becomes fifty or so.

From there on, the model is a language model in every respect that part 1 of the language models module describes. It predicts the next token, one is sampled, and the loop repeats. A decoder turns the finished sequence back into pixels or sound. Often the same transformer handles text tokens and media tokens in one sequence, so the prompt and the output share a context.

The trade

Where each leads

• Images. Diffusion dominates, and it is nearly all of what you can download. Several multimodal assistants released since 2025 appear to generate images token by token, or to combine tokens with a diffusion stage, though their makers have published little detail. At least one large open model works this way. They read prompts very well, letter text accurately and can edit a picture in conversation, because the image sits in the same context as the words.
• Video. Diffusion, almost without exception. A few seconds of video is already an enormous number of tokens.
• Music. Here the token design is strongest. Audio has a natural order and songs have structure over minutes, which is what language models are good at. The best-known song services do not publish their designs and are widely reported to work this way. Most open music models that can be run locally are diffusion. Part 4 sets them side by side.
• 3D. Diffusion for the shape. Token models have a niche in producing the tidy, low-polygon meshes an artist would make, one triangle at a time.

Hybrids

The two are increasingly combined, and the usual division of labour follows the table: a token model plans, and a diffusion model renders. A language model sketches the structure of a song and a diffusion model fills in the sound. A language model lays out a picture and a diffusion stage paints it. Expect the boundary to keep blurring. The properties in the table stay useful, because they tell you which half of a system is responsible for what you are seeing.

The rest of this module concentrates on diffusion, because that is the design you can download, adapt and run, and because its settings are the ones exposed in tools. The prompting advice in part 7 covers both.

088. What falls out of this design

Five properties follow directly from sections 1 to 7, and they hold in every medium. Parts 2 to 5 give each medium’s own version.

It models appearance, not things

The model holds no scene, no skeleton and no geometry. It has learned what images look like, to the point where light, reflection and perspective are usually convincing. But “usually looks right” is the only standard it was trained to.

This is why hands were the famous failure in images, and part 2 goes through that and its relatives. Every medium has its own. A video model has learned what motion looks like and not the physics that produces it, so a glass can refill itself and a walker’s legs can swap. A music model has learned what songs sound like and not harmony, so it cannot reliably be told a key or a chord sequence. A 3D model has learned what objects look like from outside and nothing of how they are built. Larger models and better data make each failure rarer. The cause does not go away, and it shows up wherever exactness matters and appearance is forgiving.

Every output is a fresh draw

The model keeps no state between outputs. Asking for “the same character, now sitting down” gives a new person who matches the same words. Consistency across images, shots or tracks is not something the basic machine does at all.

The techniques that supply it are the subject of part 6: reference images, image-to-image, inpainting, and small add-on models called LoRAs that teach a model one face, product or style. When a hosted assistant appears to edit a picture in conversation, it is using these techniques behind the scenes, or it is the token design from section 7.

You get the training data’s defaults

An underspecified prompt is filled in with whatever was most common in training, in any medium. A song with no stated genre comes back as glossy mid-tempo pop. A video with no stated camera comes back as a slow push-in. “A CEO”, “a nurse” or “a beautiful house” comes back as the web’s most frequent picture of each, with all the skew that implies. The same goes for aesthetics: many models have a house look, often glossy and centred, that comes from the images their makers chose to fine-tune on.

Two practical points follow. If the defaults matter, as they do for anything depicting people, specify what you want and review a batch, not a single image. And named styles work because names were in the captions. That is a legal and ethical question as well as a technical one, and part 6 covers it.

It can memorise, rarely

A model of a few gigabytes cannot store billions of images, and it does not. But an image that appeared many times in the training set, such as a famous painting, a stock photograph or a widely reposted portrait, can be learned almost exactly. Researchers who deliberately went looking, by generating many millions of images from the captions of the most duplicated training pictures, recovered over a thousand near copies across the models they tested, from photographs of individual people to company logos.

Set against the number of attempts this is rare, and removing duplicates from training data makes it rarer. It is not zero. For commercial use, a reverse image search on a final image is cheap insurance. Music is the medium where this matters most, since melodies are short, widely duplicated and fiercely protected, and part 4 returns to it.

Two machines compared

The whiteboard version

A text encoder turns the prompt into vectors. A seed makes a small grid of random noise. A large network looks at the grid, the noise level and the prompt vectors and guesses the noise, some of that guess is removed, and this repeats a few dozen times, settling layout first and detail last. A decoder expands the finished grid into pixels, frames, sound or geometry. The other design turns media into tokens and writes them one at a time, like a language model: slower, better at long structure, and ahead in music. In both, the weights never change while you use it, nothing carries over to the next output, and what you get is the most plausible-looking thing that matches your words, not a thing that is guaranteed correct.