Building on It: From Your Own Code to a Shared Service

011. Calling it from code

Part 3 described the local server and its compatible format. The practical consequence is that the client libraries published for the hosted services work against a local model when given a different address, and the higher-level libraries treat a local server as one more provider.

The practical AI module covers how to build with models: prompts as code, structured output, tools, retrieval and evaluation. All of it applies. Three points are particular to local models.

Design for the model you have

A small model follows a short, explicit prompt with one job in it far better than a long one with several. Where a hosted model would take a page of instructions and a complex task, a local one does better with the task broken into steps in your own code, each a simple call: classify, then extract, then summarise. Part 2 of the language models module makes the point that small models do well when the facts are in front of them and the task is narrow. Build for that.

Force the structure

Part 1 of the language models module explains that token choice is ordinary code, so a system can mask out any token that would break a format. The local engines expose this directly. You supply a JSON schema, or a formal grammar, and the engine permits only tokens that keep the output valid. The result always parses.

This matters more for small models than large ones, because malformed output is one of their commonest failures, and it removes that failure entirely. It guarantees the shape and not the content: a field will be present and of the right type, and may still be wrong. For extraction, classification, routing and anything feeding another program, use it by default.

Spend calls freely

Against a paid API, every call is a cost, and designs are shaped by that. Locally, calls are free and the only budget is time. That changes what is sensible.

• Try several times and check. Generate five candidates and keep the one that passes a test, or the one most of them agree on. The language models module’s idea that verification converts compute into reliability is cheapest to apply here.
• Use a second call as a checker. Ask the model, in a separate request, whether the output meets the instruction.
• Process everything. Tag every document, summarise every ticket, check every commit message. Jobs that would never justify an API bill are an overnight run.
• Run your evaluations often. Part 3 of the practical AI module says to test prompts like code. A local test set of a few hundred examples can be run on every change.

022. Your own documents

Part 8 of the practical AI module explains retrieval-augmented generation in full: split documents into passages, turn each into a vector with an embedding model so that similar meanings land close together, search by meaning and by keyword, rerank, and put the best passages in the prompt. Nothing about that changes locally. What changes is that all of it can run beside the model.

• Embedding models are small. Part 2 listed them among the things a name can tell you. Good ones have from a few hundred million to a few billion parameters, run quickly on a processor with no graphics card at all, and are published under the same licences as chat models. The local tools in part 3 serve them alongside chat models.
• The index can be a file. For thousands to a few million passages, a vector index inside an ordinary embedded database is enough. No server is needed.
• Rerankers are small too, and run locally in the same way.

This is the strongest case for local AI. A private assistant over contracts, client files, medical notes, source code or a lifetime of personal notes is exactly the thing people will not send to a third party, and it plays to a small model’s strength, since part 2 of the language models module notes that small models do well when the facts are in front of them.

Two cautions apply with more force to a small model. A small model is more easily distracted by irrelevant passages, so retrieve fewer and better ones. And part 3’s context setting applies: retrieved passages need room, and a 4,000-token default will silently discard them.

033. Other kinds of local model

These share the machine’s memory with the language model. On a single graphics card, an image model and a language model usually cannot both be loaded, and the tools swap them, which takes seconds each time. Unified-memory machines cope better, which is another point in their favour for mixed work.

Fine-tuning locally

Part 2 of the language models module explains what fine-tuning is good for, which is style, format and narrow tasks and not teaching facts, and describes LoRA, the method that makes it cheap. It is practical on local hardware. A model of 8 billion parameters can be fine-tuned with LoRA on a 16 to 24 GB graphics card in an hour or two, and the MLX tools do the same on a Mac. Variants that keep the base model quantised during training reach larger models.

It is worth considering for one situation in particular: a narrow, high-volume task where a small model is nearly good enough. A few hundred to a few thousand good examples will often take a 4 or 8 billion parameter model to the accuracy of a much larger one on that single task, and the result runs fast on modest hardware. For anything broader, a better prompt, retrieval or a larger model is the better investment, as the language models module says.

044. Serving more than one person

Why the apps do not scale

Part 1 explained that generating a token means reading the whole model from memory, and that the processor spends most of its time waiting. When several requests arrive together, a serving system reads the weights once and advances all of them at the same time. The memory is read no more often, and ten people are served in little more time than one. Part 3 of the language models module calls this batching, and explains that it is the basis of the hosted services’ economics.

The desktop apps mostly handle requests one at a time, or a few at once. One comparison published in 2026 measured a dedicated serving system at sixteen to twenty times the throughput of a desktop app under heavy simultaneous load, and found the two almost level for a single user. The gap is entirely about concurrency.

The serving systems

As of September 2026:

What it takes

• A machine that stays on, with a graphics card sized for the model plus the contexts of everyone using it at once. Every simultaneous user needs their own context in memory, so part 1’s sum is done for the peak and not the average.
• Something in front of it. Part 3 warned that these servers accept any request. A shared one needs authentication, encryption and logging, usually from a gateway placed in front.
• Someone to look after it: updates, new models, monitoring, a plan for when it is down.
• An honest estimate of use. The language models module’s conclusion about self-hosting applies at small scale too. A shared server earns its place with steady use by several people, or a continuous batch workload. For occasional use by three people, each running their own model, or an API, is simpler and probably cheaper.

The cases where a small shared server is clearly right are the ones part 1 listed: data that may not leave, a fixed model for reproducible work, and steady high-volume processing.

055. Running it responsibly

• Pin what you use. Record the exact file, its quantisation level, the tool and its version, and the settings, for anything whose results matter. Tools’ short model names are reassigned to new files without notice. This is the local form of the language models module’s rule that prompts are code and the model version is part of it.
• Know the licence, as part 2 explained, for the exact file, and keep a note of it.
• There are no guard rails unless you add them. Hosted services filter inputs and outputs. A local model does whatever it is asked. For anything user-facing that is your responsibility, and small classifier models published for the purpose, which part 2 listed under guard, are one tool for it.
• Treat model files and tools as software from the internet, because they are. Part 2’s advice on sources and formats applies, and so does keeping the tools updated: they are network services written quickly, and security fixes are frequent.
• Test before you switch. A new model with better scores may be worse at your task. Keep a small test set from your own work, as the language models module describes, and run it before changing anything.
• Review the choice regularly. Twice a year is reasonable. The best model for a given amount of memory has changed every few months, the tools’ defaults shift, and hardware that was exotic becomes ordinary.

066. When to stop

A local model is the wrong choice, and it is worth recognising quickly, when:

• The task needs the best model there is. Hard reasoning, large-scale design, long autonomous work. The language models module’s method applies: prove the task with the most capable model, then step down. If it fails at the step to local, that is your answer.
• The time spent is worth more than the API bill. For most individuals and small teams, a modest monthly spend on a hosted model costs less than the hours spent keeping a local stack running. Privacy, reproducibility or volume have to justify the difference.
• It needs to be available to many people, reliably, and nobody’s job is to keep it so.
• Nothing confidential is involved, and the vendor’s terms are acceptable, which part 3 of the AI in the organisation module explains how to check.

And it is the right choice, often the only one, when the data cannot leave, when the result must be reproducible, when the volume is large and the task is narrow, when there is no connection, or when you are building something that ships to a user’s own machine.

Because part 3’s server speaks the same format as the hosted services, this need not be decided once. Write the code against the common format, keep the address and the model name as configuration, and route by task: private and routine work to the local model, hard problems to a hosted one. That is part 2 of the language models module’s routing, with the boundary of your own machine as one of the things to route on.

The whiteboard version

A local model is called from code exactly as a hosted one is, at a different address. Build for a small model: short explicit prompts, one job per call, steps in your code and not in the prompt. Force valid structured output with a schema or grammar, which the local engines support. Calls are free, so try several times and check, and run your tests often. Private document search is the strongest use: embedding models are small, run without a graphics card, and the index can be a file. Speech recognition, image reading and image generation all run locally too, sharing the same memory. Serving a team needs a serving system such as vLLM on a Linux machine with a graphics card, sized for everyone’s contexts at once, behind authentication, and only steady use justifies it. Nobody stands behind a local model, so pin the exact file and settings, know the licence, add your own limits, and review the choice twice a year. Use local for the private, the routine, the bulk and the offline, a hosted model for the hardest work, and write the code so that either can answer.