Running a Model: Engines, Apps and the Settings That Matter

011. Engines and layers

A useful summary from one 2026 comparison: some of these tools are experience layers, some are engines, and one is a serving system. They are less competitors than different layers of one stack.

The engines

llama.cpp is the engine under most local software. It is an open source project written in C and C++, which created the GGUF format that part 2 described. Its distinguishing quality is that it runs on anything: NVIDIA, AMD and Intel graphics cards, Apple silicon, ordinary processors with no graphics card at all, phones and single-board computers. It supports every quantisation level in part 2, and it can split a model between a graphics card and ordinary memory when it does not fit. It ships with its own server and a simple web page for chatting, and it can be used directly. New model designs usually work in llama.cpp within days of release, and occasionally take weeks.

MLX is Apple’s framework for running models on Apple silicon, and the fastest way to do so. Measurements published in 2026 put it at around one and a half times the speed of llama.cpp on the same Mac for dense models, and more for mixture-of-experts models. It runs only on Macs, and uses its own model format, which part 2 described.

That is nearly the whole list. On a Mac, the engine is MLX or llama.cpp. On anything else, it is llama.cpp, until you are serving a team, when part 5’s serving systems take over.

022. The apps

As of September 2026:

How to choose

If you write software, start with Ollama: it is what the rest of the ecosystem assumes. If you do not, or you want to see what you are doing, start with LM Studio. Move to llama.cpp itself when you hit something the apps do not expose. On a Mac, check that whichever you choose is using MLX.

They coexist without trouble, and a model downloaded for one can usually be pointed at by another, though each keeps its files in its own place and you can easily end up with the same 15 GB file three times.

Names within names

One source of confusion is worth knowing in advance. Ollama and similar tools keep their own catalogues, in which a model has a short name and a tag, such as a family name followed by a size. That short name stands for one particular conversion at one particular quantisation level, usually 4 bits, chosen by the tool. It is not the full name that part 2 taught you to read, and two tools’ short names for “the same” model may be different files. When it matters, look at what the short name resolves to. Both Ollama and LM Studio can also fetch any GGUF file from Hugging Face by its full name.

033. The settings that matter

Context length

Part 3 of the language models module explains the context window and the KV cache that holds it. A model card will advertise a window of 128,000 or 256,000 tokens. What you get when you run it is whatever the software allocates, which is a separate and much smaller number.

Ollama’s documentation states its defaults: 4,000 tokens on a machine with under 24 GB of graphics memory, 32,000 with 24 to 48 GB, and 256,000 with more. Four thousand tokens is about 3,000 words. A single source file, a few turns of conversation and the instructions a coding tool sends before you type anything all exceed it.

What happens next is the important part. The software does not report an error. It discards the oldest text and carries on, and the model answers from what is left. The symptoms look like a stupid model: it forgets the instructions, ignores the file it was just given, loses track after a few turns, or fails to use its tools. The same documentation recommends at least 64,000 tokens for agents, coding tools and web search. Every harness in part 4 says the same.

The setting costs memory, as part 1 explained: tens to hundreds of kilobytes per token, depending on the model. On a 24 GB card with a 16 GB model, a 64,000-token context may not fit at full precision, which leads to the next setting.

• Check the context length before judging any model. It is the first thing to look at when a local model seems worse than its reputation.
• Set it to what the job needs, and no higher. Memory reserved for context you do not use is memory a better model could have had.
• Check it took effect. The tools will show how much memory a loaded model is using and whether it is on the graphics card.

Fitting it on the graphics card

An engine loads a model layer by layer, and each layer goes either to the graphics card or to ordinary memory. Part 1 explained the cost: anything not on the card runs many times slower.

The apps choose automatically, and a model that mostly fits will be split without telling you. If generation is much slower than part 1’s sum predicts, this is the usual reason. The tools report the split. The fix is a smaller model, a lower quantisation level from part 2, a shorter context, or the next setting.

For mixture-of-experts models there is a better split than layer by layer, which part 1 mentioned: keep the parts every token uses on the card and the experts in ordinary memory. llama.cpp supports this, and it makes models of 30 billion parameters and more usable on a 16 GB card.

Squeezing the context

Two settings reduce what the context costs.

• Flash attention is a faster, leaner way of computing attention that gives the same results. Most tools now turn it on by default. If yours has not, turn it on.
• Quantising the KV cache stores the context at 8 bits in place of 16, halving its memory for a loss in quality that is usually too small to notice. Four bits is available and does cost quality. At 8 bits this is often what makes a long context fit.

The prompt template

Part 1 of the language models module explains that a chat is one long token sequence with special markers between turns. Every model family uses different markers, and a model only behaves as trained when it sees its own.

A GGUF file carries its template inside it, and the apps apply it for you, so this normally just works. It goes wrong in three situations: a conversion made with a faulty template, which the community usually finds and fixes within days of a model’s release, which is a reason not to download on day one; a tool that overrides the template with its own; and tool calling, which part 4 covers, where the template also defines how the model asks to use a tool. The symptom is subtle. The model works, and is noticeably worse than it should be, or never calls a tool.

Sampling

Temperature and the related settings are explained in part 1 of the language models module. The practical point for local use is that model cards now state recommended values, which differ between families and between a model’s thinking and non-thinking modes, and the models are sensitive to them. A model that rambles, repeats itself or loops is often running with the wrong ones. Use the card’s.

044. The local server

The hosted services settled, more or less, on one format for requests and responses, first defined by OpenAI’s API: a list of messages in, a message out, with optional tool definitions and streaming. llama.cpp, Ollama, LM Studio, MLX and vLLM all provide a server that accepts that format on the local machine.

So a local model is, to other software, just another provider. The harnesses in part 4, editor extensions, libraries and your own scripts need an address and a model name. Nothing else changes. This is also how the two ways of working combine: the same code can use a local model for private or routine work and a hosted one for the hard cases, as part 5 describes.

Four cautions.

• “Compatible” is approximate. The basic chat format works everywhere. Tool calling, structured output, images and reasoning fields vary between servers and between models, and a newer format from one vendor is supported by some local servers and not others. When a tool misbehaves with a local model, this is the second thing to check, after the context length.
• The server is unprotected. By default these servers listen only on the local machine, accept any request and need no password. That is fine. Making one reachable from the network, which is one setting, exposes an open, unauthenticated service. Anyone who can reach it can use it and, depending on the tool, load and delete models. Put it behind something that checks who is asking, or do not expose it.
• One request at a time, by default. The apps are built for one user. Part 5 covers serving several.
• Loading takes time. The first request after a model is loaded waits while gigabytes are read from disk. The apps unload idle models after a few minutes, and a slow first answer is that and not a fault.

055. Knowing how fast it is

The tools all report, for each response, the prompt processing speed and the generation speed, both in tokens per second. llama.cpp includes a benchmarking tool that measures both under controlled conditions, and the apps show them in the chat window or a verbose mode.

• Generation speed should be within sight of part 1’s ceiling: memory bandwidth divided by the size of the weights read for each token. Half to three quarters of the ceiling is normal. A fifth of it means part of the model is in ordinary memory.
• Prompt processing speed is usually ten to a hundred times the generation speed, and is what sets the wait before the first word. For an agent that sends 30,000 tokens of instructions and files with every request, it matters more than generation speed.
• Both fall as the context fills, since there is more to attend to. Measure at the context length you will use.
• Thinking models hide their cost. A model that writes two thousand tokens of reasoning before a fifty-token answer has, at thirty tokens a second, kept you waiting over a minute. For interactive use, a non-thinking mode or a faster model is often the better trade.

Useful speeds, as a rough guide: ten tokens a second is reading speed and fine for chat. Thirty is comfortable. Agents want fifty or more, along with prompt processing in the thousands, because they generate and re-read a great deal of text that no person reads.

The whiteboard version

There are two engines that matter locally: llama.cpp, which runs GGUF files on anything, and MLX, which is faster on Macs. Ollama and LM Studio are apps built on them, one for the command line and one with a window, and both can serve a model on your own machine in the format the hosted services use, so any tool can be pointed at it. Three settings decide whether it works. Context length defaults to as little as 4,000 tokens, and text beyond it is silently discarded: set it to what the job needs, 64,000 for agents, and check the memory. Make sure the model is actually on the graphics card. And make sure the prompt template is the model’s own. Quantise the KV cache to 8 bits if the context will not fit. Then measure both speeds against what the hardware should deliver.