Shaping Models: Fine-Tunes, LoRAs, Style and Control

011. The open ecosystem

Base models and families

A base model is one trained from scratch by a lab, at a cost of hundreds of thousands to many millions of dollars. Few organisations do this. Everyone else starts from a base model that someone has published. Parts 2 to 5 list the current families for images, video, music and 3D.

Add-ons belong to a family

A LoRA, a control model or a fine-tune is made for one base model and works only with that model and its descendants. An add-on for Stable Diffusion 1.5 does nothing useful on SDXL. An add-on is a set of adjustments to particular weights, and it means nothing applied to different ones.

So the choice of family is a choice of ecosystem. The newest base model often produces the best raw images and has the fewest add-ons. SDXL, three years old, still has by far the largest library. For a job that needs a specific control or style, the older family is often the right answer, and for a job that needs lettering or a long, exact description, it never is.

What you are actually downloading

Models are shared on public hubs as single files, and three facts about those files matter to anyone responsible for a team’s machines.

• They are large, and growing. A first-generation model is about 2 GB. SDXL is about 6.5 GB. A 12 billion parameter model is over 20 GB at full precision and a 32 billion parameter one over 60 GB, before counting a text encoder that may be a large language model in its own right. These are usually run in an 8-bit or 4-bit quantised form to fit on one graphics card, with the same trade of a little quality for a lot of memory that part 3 of the language models module describes.
• The old file format can run code. Early checkpoints used a general-purpose Python format in which loading a file can execute whatever the file’s author put there. The replacement, safetensors, holds only numbers and cannot. Treat any model file that is not safetensors the way you would treat an unknown executable, and do not load one on a work machine.
• Anyone can upload. Hubs host hundreds of thousands of community files with little review. Provenance, licence and content vary from careful to absent. Section 9 returns to this.

The tools

Open models are run through community software, and the two main styles suit different people. Form-based interfaces present the settings from part 1 as boxes and sliders. Node-based interfaces present the pipeline as a graph of connected blocks: load model, encode prompt, sample, decode. The graph looks forbidding and is the better choice for anything repeatable, because the whole pipeline is a file that can be saved, shared, versioned and rerun. Engineers should recognise it as a build definition.

For integration into a product there are code libraries that expose the same pieces, and hosted inference services that will run an open model, with your add-ons, behind an API.

022. Fine-tunes and checkpoints

A checkpoint is a complete set of model weights in one file. A full fine-tune takes a base checkpoint and continues training all of its weights on a new, narrower set of captioned images, using exactly the noise-removal game from part 1. The result is a new checkpoint of the same size.

Communities have produced thousands of these, each pulling a base model towards one look: photorealism, a style of illustration, product shots, pixel art. For the first two generations of Stable Diffusion, nearly all serious work was done on a community fine-tune and not on the base model, because the fine-tunes were simply better within their niche.

The cost is breadth. Training on a narrow set overwrites some of what the model knew, and a checkpoint tuned on portraits may have lost most of its ability to draw a landscape. The same happens when a language model is fine-tuned, and the consequence is the same: a fine-tune is a specialist.

Merges

Because two fine-tunes of the same base have the same shape, their weights can be averaged. A merge of a photographic checkpoint and an illustrative one gives something in between, with no training at all. Most popular community checkpoints are merges of merges.

This matters for one reason beyond curiosity. A merged checkpoint has no clean record of what it was trained on or what licences its ancestors carried. For hobby use that is unimportant. For commercial use it can be disqualifying.

When a full fine-tune is the right tool

Rarely, for most organisations. It needs thousands of well-captioned images, real GPU time and someone who knows how to avoid wrecking the model. It makes sense when you need a broad, consistent house look across many subjects, and you have the image library to teach it: a game studio’s art direction, a publisher’s illustration style, a retailer’s product photography.

For one character, one product or one style, a LoRA does the job at a hundredth of the cost, and section 3 is about that.

033. LoRAs

The idea

Low-rank adaptation was invented in 2021 for large language models and adopted by the image community within months of Stable Diffusion’s release. Part 2 of the language models module describes it in a deep dive, and the mechanism here is identical.

A full fine-tune changes every number in every weight matrix. LoRA makes a bet: the change needed to teach a model one new concept is simple, and can be captured by a much smaller set of numbers. So the original weights are frozen, and beside each of the chosen matrices, usually those in the attention layers, training learns a small correction. At generation time the correction is added to the original weights.

Three properties follow, and they are what made LoRAs take over.

• They are small. Typically 10 to 200 MB, against several gigabytes for the model. A library of hundreds is practical.
• They have a strength dial. Because the correction is added, it can be added at half strength, or at one and a half. Most tools write this in the prompt, as a name and a number.
• They stack. Two or three can be applied at once: a character, a style and a lighting look. They interfere with each other as the count and strengths rise, so stacking is a craft and not a guarantee.

What people train them on

A subject LoRA is normally trained with a trigger word, a rare token that stands for the subject in captions. Put the trigger in a prompt and the subject appears. Leave it out and the model behaves nearly as before.

The training set is the whole job

LoRA training is ordinary supervised learning on a tiny dataset, and everything the language models module says about data quality applies with more force. A few rules account for most of the difference between a LoRA that works and one that does not.

• Vary everything except the thing you are teaching. If every photo of the product is on a white table, the model learns that the product includes a white table. Different backgrounds, angles, lighting and distances teach it what is constant.
• Caption what you want to stay changeable. Anything described in a training caption is attributed to the caption’s words. Anything left out is absorbed into the trigger. Caption the background and the pose, and they remain under your control. Leave out the jacket the character always wears, and the jacket becomes part of the character.
• A few good images beat many poor ones. Blurry, cropped or watermarked images teach blur, crops and watermarks.
• Stop early. An overtrained LoRA reproduces its training images and resists the prompt. Trainers save a copy every few hundred steps so you can pick the last one that still listens.

What goes wrong

• It bleeds. A style LoRA trained on portraits makes everything a portrait. A character LoRA pulls every face towards the character. Lower the strength before anything else.
• It fights the base. A LoRA trained on one fine-tune often works on its siblings and sometimes does not. Test on the checkpoint you will actually use.
• It quietly carries its data. A LoRA trained on an illustrator’s portfolio reproduces that illustrator’s style on demand. A LoRA trained on photos of a real person produces that person doing anything at all. Both are trivially easy, and section 9 explains why an organisation needs a rule about each.

What hosted services offer instead

Hosted services rarely let you upload a LoRA, but many offer the same outcomes under other names: “custom models” or “brand styles” trained from your uploads are usually LoRAs behind the scenes, and “character reference” and “style reference” features are the adapter techniques in section 6. The service chooses the settings. You give up control and portability, and gain not having to run anything.

W_used = W + strength * (B @ A)   # W: 4096 by 4096.  B: 4096 by r.  A: r by 4096

044. Starting from an image

Image-to-image

Part 1 explained that a picture forms coarse to fine: layout in the early steps, detail in the late ones. Image-to-image exploits that directly.

Encode your starting image into a latent. Add noise to it, but not all the way: stop at a chosen point part way up the noise scale. Then run the ordinary loop from that point down. The model treats your noised image as if it were its own half-finished work, and finishes it according to the prompt.

The one setting that matters is how far up the scale you go, called denoising strength, from 0 to 1.

A sketch, a photograph of a whiteboard, a screenshot of grey boxes from a layout tool or a crude 3D render all work as starting images. For many design tasks this is the best trade in the whole field: a person supplies the composition, which models are poor at following from words, and the model supplies the rendering, which is slow for a person.

Inpainting and outpainting

Inpainting redoes part of an image and leaves the rest untouched. You paint a mask over the area to change and write a prompt for what should be there. During the loop, the area outside the mask is reset at every step to the original image at the matching noise level, so the model is constantly shown the true surroundings and paints the masked area to fit them.

Outpainting is the same operation with the mask outside the frame: extend the canvas, mask the new empty area, and the model continues the scene.

Three practical points save most of the frustration.

• The model only knows what it can see. Tools usually crop to a region around the mask for speed. If the crop excludes the light source or the rest of the figure, the fill will not match. Widen the context area.
• Seams come from hard masks. Feather the mask edge and let it overlap what you are keeping by a few pixels.
• Dedicated inpainting models do it better. These are versions of a base model trained with the mask and the surrounding image as extra inputs. They fill more coherently than a standard model used with the reset trick.

Inpainting is how professionals fix the failures that part 2 listed. A wrong hand, garbled lettering or an extra finger is masked and regenerated, several times if need be, while the ninety-five percent of the image that was right is kept.

Editing by instruction

A newer kind of model takes an image and a written instruction, “make it night”, “remove the car”, “put her in a red coat”, and returns the edited image with no mask. Early versions from 2022 were unreliable. Open and hosted editing models released since 2025 are good enough for routine work, and they keep a face or a product recognisable across edits far better than image-to-image does.

They work by giving the denoiser the original image as an additional input alongside the noise, and training on large sets of before-and-after pairs. When a hosted assistant edits a picture in conversation, this, or the token-based design from part 1, is what is happening.

Treat instruction editing as the first thing to try for a simple change, and masks as the tool for when it gets the area or the extent wrong.

055. Controlling structure

Part 2 showed that composition is the weakest thing to steer with a prompt: counts, positions and relationships all go astray. Image-to-image helps but carries colour and texture along with layout. Control models separate the two. They take a structural map of an image and make the generated image conform to it, leaving appearance entirely to the prompt.

The best-known method, ControlNet, was published in early 2023. Each control model is trained for one kind of map.

The maps are made by small preprocessing models that ship with the tools: one click turns a photograph into its pose, edges or depth. So a common workflow is to find or shoot a reference with the right structure, extract the map, and generate something entirely different on top of it.

Two settings matter. Control weight sets how strictly the map is followed. Start and end points set which part of the step range it applies to. Because structure is settled in the early steps, a control applied only for the first third or half of the steps fixes the layout and then lets the model finish freely, which usually looks more natural than full control throughout.

Several controls can be combined, such as pose and depth together, at the cost of memory and time. Control models are tied to a base model family, as every add-on is, and the older families have by far the widest selection.

Hosted services expose little of this. Some accept a “structure reference” or a sketch. If a job depends on exact layout, that alone can decide between hosted and local.

066. Style transfer

Two things with one name

Neural style transfer, from 2015, takes one content image and one style image and optimises a new image until its large-scale structure matches the first and its texture statistics match the second. It involves no text and no diffusion. It produced the “photo as a Van Gogh” filters of the following years, and it copies brushwork and palette well and understanding not at all: it cannot redraw a face the way an artist would, only repaint the photograph in their textures.

What people mean today is broader: produce new images, of any subject, that belong to a given visual style. A diffusion model can do that because it has learned styles as concepts, and there are four routes to it.

Four routes

Reference adapters

The third route deserves a little more, because it is new since the first generation and changes what is possible without training.

Part 1 described how the denoiser consults the prompt through cross-attention. An image prompt adapter adds a second, parallel channel. A reference image is passed through an image encoder, usually the image half of CLIP, and the resulting vectors are offered to the denoiser through additional attention layers trained for the purpose. The model can then be steered by a picture in the same way it is steered by words, and by both at once.

One reference image can stand in for a paragraph of description that no one could write. The same mechanism, trained on faces, gives the “character reference” features that hold a person’s appearance across images without a LoRA. The limits follow from the encoder: it captures the overall look, the palette and the subject, and loses fine detail, so a reference gets you a family resemblance and a LoRA gets you the thing itself.

Choosing between them

Start at the top of the table and move down only when the result is not close enough. A described style costs nothing to try. A reference image is the right tool for one-off work and mood exploration. A LoRA is justified when the style must be held across hundreds of images and many people, which is the definition of a brand style.

The harder question is whose style it is. All four routes can imitate a living artist, and the last does it best. Section 9 deals with that.

077. The ladder in other media

Three things are worth drawing out.

• Starting from media is the normal method everywhere except images. Parts 3 and 5 both recommend making a still first and generating from it. The still carries the composition and the consistency, which means the image techniques in sections 2 to 6 are how you control video and 3D as well.
• LoRAs carry over where there is an open family to attach them to. Open video models have growing libraries of them. Training one for video needs far more memory and time than for images, and clips in place of pictures. For music they exist on open models and raise section 9’s questions in their sharpest form, since a LoRA of one artist’s catalogue imitates a living performer.
• The rungs are merging into the models. Part 2 noted that current image models generate, edit and follow references in one system. The same is happening in video, and it is how hosted services expose these abilities at all. The ladder remains the right way to think, since it tells you what kind of input will fix what kind of problem, even when all of it sits behind one prompt box.

088. Local or hosted

What local needs

An image model needs a graphics card with enough memory to hold the weights and the working data. The figures below are illustrative and assume a consumer card.

Laptops and desktops that share memory between processor and graphics can run the larger models, more slowly. Each control model or additional encoder adds to the memory needed. For a team, the usual arrangement is one shared workstation or a rented cloud GPU with a web interface, not a powerful card on every desk.

The comparison

The balance differs by medium, as parts 3 to 5 found. In video and in song generation the hosted services are well ahead and the open models demand serious hardware, so local is a deliberate choice made for control or privacy. In 3D the open models are close to the front. For sound effects and instrumental music, open models are small and easy to run.

How the decision usually goes

Hosted is the default, for the same reasons an API is the default for language models: no operations, the best quality, and someone else carries the upgrade burden. Most marketing, presentation and concept work needs nothing more.

Local earns its place when one of four things is true.

• The work needs control that hosted does not expose. A consistent character across a hundred frames, a product that must be exact, layouts that follow a wireframe.
• The inputs cannot leave. Unreleased products, client material, anything under a confidentiality agreement.
• The output must be reproducible. Regulated or audited work, or a production pipeline that cannot tolerate the model changing mid-project.
• The volume is high and steady. Catalogue imagery, game assets, personalised creative at scale. At thousands of images a day the arithmetic reverses.

Unlike a large language model, an image model that is good enough for production still fits on one consumer card, even though the largest no longer do. The self-hosting sums that the language models module found so unfavourable are far kinder here, and that is why a do-it-yourself ecosystem thrives for images in a way it does not for the largest language models.

099. Rights, likeness and provenance

Training data

The first open models were trained on billions of images scraped from the web without the permission of the people who made them. Whether that is lawful is being fought over in several countries, and the answer differs by jurisdiction and is not final in any of them.

One decided case shows how narrow the answers so far have been. In November 2025 the High Court in London ruled on a claim by a large picture agency against the maker of Stable Diffusion. The agency abandoned its central claim, that training on its images infringed copyright, because it could not show that the training had happened in the United Kingdom. The court then rejected the argument that the model is itself an “infringing copy” of the images it was trained on, observing that the model does not store them. The agency won on one point, that some early versions had generated images bearing its watermark, which the judge described as historic and extremely limited in scope. The main question was never reached.

Cases in the United States, including claims by artists and by film studios, are continuing. The practical position for an organisation is a choice of risk, not a settled rule.

• Lowest risk: a hosted service from a vendor that trains only on licensed and public-domain images and offers business customers an indemnity.
• Moderate: a major hosted or open model with a known publisher, used for ordinary purposes.
• Highest: an anonymous community merge of unknown ancestry, or a LoRA trained on one living artist’s work and used to produce work in competition with theirs. Whatever the courts decide about training in general, that last case is the one most likely to end badly, legally and reputationally.

Who owns the output

In the United States, a work must have a human author to be protected by copyright. The Copyright Office’s position, set out in a report in January 2025, is that an image produced from a prompt alone is not protected, however elaborate the prompt, because the user does not control how the model turns it into an image. Human contributions are protected where they exist: a selection and arrangement, substantial editing, a drawing that the model rendered. The United Kingdom and some other countries treat computer-generated works differently, and the position there is under review.

The consequence is easy to miss. If your campaign image came straight from a prompt, a competitor may be free to copy it. For anything that matters commercially, make sure a person does real creative work on the image, through composition, control images, painting over or compositing, and keep the working files that show it.

Model licences are a separate matter. “Open” covers everything from permissive licences that allow any use, through licences that forbid listed harmful uses, to licences that are free only for non-commercial use or only for companies below a revenue threshold. Different sizes of the same model often carry different licences. Read the licence of the exact file, and of every add-on, before commercial use.

Music

Recorded music is the exception to almost everything above, because it is owned by a few large companies that acted together. Part 4 covers the lawsuits, the settlements of late 2025 and what they mean for using a generated song. The short version is that what you may do depends on the service and on your plan, and that resemblance to an existing tune is a more serious risk than resemblance to an existing picture.

Likeness

A subject LoRA of a real person takes twenty photographs and an hour. It will then produce that person anywhere, doing anything. Reference adapters do nearly as well with one photograph. A voice can be copied from under a minute of recording, as part 4 describes, and part 3 noted that video models now generate speech with matching lip movement, so a convincing clip of a real person saying something they never said is a single prompt.

Non-consensual sexual imagery of real people made this way is a crime in a growing number of jurisdictions and a serious harm everywhere. Fake images of public figures are a fraud and disinformation tool. Even benign uses, such as an employee’s likeness in marketing, engage rights of publicity and data protection law, since a face is biometric personal data.

Hosted services block much of this with filters. A local model has no filters at all, so the only control is the organisation’s own. A workable rule is short: no model, LoRA, reference or voice clone of a real person without their written consent for the specific use, and no exceptions for “internal” or “just testing”.

Provenance

Part 2 argued that no one can reliably tell a generated image by eye, and the same now goes for a voice and for a few seconds of video. The alternative is to record where an image came from at the moment it is made, in a way that can be checked later.

The industry standard for this is Content Credentials, from a coalition called C2PA. A credential is a signed record attached to the file: what made it, when, and what edits followed, each step signed by the tool that performed it. Cameras from several manufacturers, the major creative software suites and most large hosted image generators now attach them, and some platforms display them.

Three limits keep expectations honest.

• They are easily lost. A screenshot, or an upload to a platform that strips metadata, removes the credential. Its absence therefore proves nothing.
• They record, they do not judge. A credential says “this tool made this”, not “this is true”.
• Open models attach nothing unless you make them. A locally generated image is, by default, unlabelled.

Some hosted generators also embed an invisible watermark in the pixels themselves, which survives cropping and compression better than metadata does. Only the vendor’s detector can read it, and open models have nothing equivalent that cannot be switched off.

Labelling is also becoming a legal duty. The EU AI Act’s transparency duties, which have applied since 2 August 2026, require that AI-generated or manipulated images that could pass as real are disclosed as such. Part 3 of the AI in the organisation module covers the Act and its dates.

For your own work the habit is simple and cheap: keep, for every published image, the model and version, the add-ons, the prompt, the seed and settings, the source images and the name of the person who made it. Tools for open models already write most of this into the file. It is your evidence of authorship, your answer to a rights query, and your way of making the image again.

1010. Choosing the lightest tool

The ladder from the start of this part, as a decision table:

Three rules sit behind the table.

Move down a rung only when you can say what the rung above failed at. “It does not look right” is a reason to improve the prompt. “The face changes between images” is a reason for a reference or a LoRA. Naming the failure tells you which tool addresses it.

Put people where models are weak. Composition, exact products and lettering are cheap for a person and unreliable for a model. Rendering, lighting, texture and variation are the reverse. The strongest workflows hand each side the work it is good at: a person blocks out the layout and supplies the real logo, the model renders the scene, and a person fixes what is left.

Treat a working setup as an asset. A model file, its add-ons and their strengths, a prompt template, the settings and a control workflow together are a piece of production tooling. Pin the versions, store them with the project, and record them with every image. This is the image counterpart of the rule in part 3 of the practical AI module that prompts are code.

The whiteboard version

A base model can be pushed towards what you need at several levels. Words and seeds cost nothing and cannot hold a subject or a layout. A starting image or a control image fixes the layout and leaves the look to the prompt. A reference image lends a look or a face without training. A LoRA, a small add-on trained on a few dozen images, teaches one subject or style and holds it reliably. A full fine-tune changes the whole model for a broad house look. Hosted services are easier and better out of the box, and open models are the ones you can shape, pin and keep private. Whichever you use, know the licence, never model a real person without consent, make sure a person’s creative work is in anything you need to own, and keep the record of how each image was made.