What You Need: Memory, Bandwidth and the Machine on Your Desk

011. Should you

The language models module reached a blunt conclusion about self-hosting: for nearly every organisation, a rented server for a large open model sits idle most of the day and costs more than the API it replaces. That conclusion stands. This module is about something smaller and more personal: a model on your own laptop, workstation or a box under the desk, serving one person or a small team.

What you gain

• Privacy that does not depend on a contract. Nothing leaves the machine. For source code under a confidentiality agreement, client documents, medical or legal material and anything a contract forbids sending to a third party, this is the whole argument.
• A model that does not change. A file on your disk behaves the same next year. Hosted models are revised and retired on the vendor’s schedule, and a prompt tuned for one version may not survive the next.
• No meter. Once the hardware is bought, a million tokens cost the electricity. Experiments you would never run against a paid API become free to try, and that changes how you work.
• It works offline, on a train, on a plane and during an outage.
• Understanding. Nothing teaches what a context window, a quantisation level or a token per second really is like watching one run out.

What you give up

• Quality. The best open models have hundreds of billions to trillions of parameters and need a rack of data-centre GPUs. What fits on a desk is the 4 to 35 billion parameter range, with a few larger exceptions on expensive machines. Those models are very good and they are not the best. The gap is widest on exactly the hard, long, multi-step tasks where a model is most valuable, as part 4 shows.
• Speed at the top end. A hosted service answers from hardware costing hundreds of thousands of dollars. A laptop does not.
• Your time. Choosing, downloading, configuring and updating is work, and the defaults are often wrong in ways part 3 describes.
• Someone else carrying the risk. No vendor filters the output, indemnifies you or fixes the security hole.

When local wins

For everything else, which is most things, the language models module’s advice holds: use the most capable hosted model that your tests justify. Many people end up using both, a local model for private and routine work and an API for the hard problems.

022. Memory decides what runs

The sum

Part 1 of the language models module describes a model as billions of numbers called weights, and its part 3 describes quantisation: storing each weight in fewer bits, trading a little quality for a lot of memory. Put together, they give the size of the file.

memory for the weights  =  parameters  x  bits per weight  /  8

Four to five bits per weight is where most people settle. Part 2 explains the names these compressed versions go by and what each costs in quality.

The weights are not everything

On top of the weights you need room for three more things.

• The context. The KV cache, which the language models module explains, holds the conversation so far and grows with every token. For a mid-sized model it is somewhere between 50 and 300 kilobytes per token, so a 32,000-token context can need several gigabytes and a 128,000-token one can need more than the weights. Part 3 shows that this is the setting most often got wrong.
• Working memory for the computation itself: allow a gigabyte or two.
• Everything else on the machine. On a computer where the model shares memory with the operating system, the browser and the editor, those come out of the same budget.

A workable rule: the weights should take no more than about two thirds to three quarters of the memory available to the model. A 15 GB file is comfortable in 24 GB and will not run usefully in 16 GB.

Two kinds of memory

Where that memory is matters as much as how much there is.

• A graphics card has its own memory, called VRAM, which is very fast and strictly limited: 8 to 32 GB on consumer cards. The model must fit in it to run at full speed. If it does not, the overflow goes to the computer’s ordinary memory and runs on the processor, and speed falls by a factor of five to twenty for that part.
• Unified memory is one pool shared by processor and graphics, as on Apple silicon Macs and some recent AMD and NVIDIA machines. There is much more of it, up to 128 GB or beyond, and all of it is available to the model. It is slower than a graphics card’s memory, for a reason the next section gives.

This is the central trade in choosing local hardware. A graphics card is fast and small. Unified memory is large and slower. Which suits you depends on whether you would rather run a 14 billion parameter model very quickly or a 70 billion parameter one at reading speed.

033. Bandwidth decides how fast

Why reading speed is the limit

The language models module explains that generating each token takes one full pass through the network, and that during generation the processor spends most of its time waiting for weights to arrive from memory. The arithmetic is simple. If the file is 15 GB and the memory can deliver 300 GB a second, the weights can be read at most twenty times a second, so twenty tokens a second is the ceiling whatever the processor.

tokens per second, at most  =  memory bandwidth  /  size of the weights read for each token

For a dense model the weights read for each token are the whole file. For a mixture-of-experts model they are only the active part, which is why such models feel so much faster than their size suggests. Real speeds come in at perhaps half to three quarters of the ceiling, and fall as the context fills.

Reading speed is comfortable from about ten tokens a second. Thirty feels quick. Agents, which part 4 covers, want much more, since they generate a great deal of text nobody reads.

Two phases, two limits

There are two phases, and different hardware is good at each.

• Reading the prompt processes all the input tokens at once. This is limited by raw computing power, where graphics cards are far ahead. It decides how long you wait before the first word appears.
• Writing the answer produces one token at a time. This is limited by memory bandwidth, as above.

For chat the first phase hardly matters, since prompts are short. For agents and for work over long documents it dominates: a 50,000-token context takes a few seconds to read on a high-end graphics card and can take minutes on a machine with slow prompt processing. This is the unadvertised weakness of large unified-memory machines, and it is improving: Apple’s newest chips add hardware specifically to speed it up.

What this explains

• A five-year-old graphics card can beat a new laptop processor, because its memory is several times faster.
• Processor cores, clock speed and “AI TOPS” figures tell you almost nothing about language model speed. Advertised neural processing units in laptops are, at the time of writing, little used by the software in part 3.
• Ordinary desktop memory is the slow lane. A desktop computer’s main memory delivers 50 to 100 GB a second, a tenth or less of a good graphics card. A model running from it manages a few tokens a second.
• Two graphics cards pool their memory and not their speed. Software can split a model across two cards, which lets a bigger model fit. Each token still passes through them in turn, so it runs at about the speed of one.

044. The machines

Figures are manufacturers’ published specifications as of September 2026.

The three platforms

• NVIDIA. Everything in part 3 works, first and best. Its programming platform, CUDA, is what nearly all AI software is written for, and new models and techniques arrive here first. The limit is memory per dollar: consumer cards stop at 32 GB.
• Apple. A Mac is the simplest way to get a lot of fast-enough memory, which makes it the common local machine among engineers. Apple’s own framework, MLX, is now the fastest way to run models on it, and the main tools use it. It is slower than a graphics card at reading long prompts, and it does not run the server software that part 5 describes for serving a team.
• AMD. Its graphics cards and unified-memory machines offer the most memory for the money. Its programming platform, ROCm, works with the main engines on Linux and increasingly on Windows, and a cross-vendor alternative, Vulkan, works nearly everywhere and is slower. Expect to spend more time on setup, and check that the specific tool you want supports the specific card you have.

Choosing

Decide what you want to run before you look at machines.

1. Pick the model size your work needs. Part 2 helps with this. For private chat, summarising and simple coding help, 8 to 14 billion parameters is enough. For serious coding assistance and agents, the 27 to 35 billion range is where models become dependable.
2. Work out the memory from section 2, with room for the context you need.
3. Decide what speed you can live with from section 3. Chat is forgiving. Agents are not.
4. Then choose the platform that delivers both within your budget.

And before buying anything, try it. Whatever machine you already have will run a 4 billion parameter model well enough to learn everything in parts 2 and 3, and an hour’s rental of a cloud GPU will tell you what a larger one would feel like.

The whiteboard version

A model is a file of numbers that must sit in fast memory while it runs. Its size is its parameter count times the bits per weight, so 27 billion parameters at the usual 4 to 5 bits is about 15 GB, plus room for the context. Memory capacity decides what runs at all. Memory bandwidth decides how fast, because every token means reading the active weights once: bandwidth divided by file size is the ceiling on tokens per second. Mixture-of-experts models need memory for all their parameters and run at the speed of the few they use. Graphics cards are fast and small, unified-memory machines are large and slower, and ordinary memory is the slow lane. Local is for private data, reproducible work, steady volume and learning. For the hardest problems, the best models are still somewhere else.