Practical AI · Part 9 of 9

Guardrails and Evals

  • About 25 minutes
  • Written 19 September 2026 by Chris Neale

After this part you can

Contain what a model can do, and measure whether it is doing it well.

About this part

This is the last of nine parts in the practical AI module. The aims of the guide, the layout every part follows and suggested reading routes are in Introduction. The format is the same as elsewhere: In plain terms opens each numbered section, deep dives are optional, and a glossary closes the part. Reading time is about 25 minutes, not counting the reference sections.

Parts 3 to 8 built a system: instructions, the files that extend an agent, the agent itself, media, tools and retrieved documents, all feeding a model that acts. This part is about trusting it. Guardrails limit what can go wrong. Evals measure how often it goes right. Section 3 condenses the whole module into a checklist for reviewing any AI feature, and the part closes with two reference sections for the module. Part 1 of the AI in the organisation module then applies all of it along the whole software lifecycle.

What part 9 gives you

Part 9 builds one idea: you cannot make a model reliable, and you can make a system reliable. Do not depend on the model behaving well. Arrange things so that misbehaviour cannot do much harm, and measure how often the whole thing succeeds on real cases. Both are ordinary engineering, and both are what allow a person to step out of the loop, which is where the speed is.

1. Guardrails

In plain terms

Guardrails are the controls around a model that limit the damage when it is wrong or has been manipulated. The principle is not to rely on the model behaving well, but to design things so that misbehaviour cannot do much harm. The methods are those of ordinary security: minimal permissions, approval for risky actions, isolation and logging.

Who should read it: everyone. Anyone approving an AI rollout should be able to ask about each of these.

One guardrail per failure

Failure from part 3 of the language models moduleGuardrail
HallucinationGround answers in sources, check citations, verify by running things
Non-determinismValidate output, retry with the error, limit steps and time
Prompt injectionLeast privilege, approval gates, isolation of untrusted content, sandboxing
Stale knowledgeSupply current documentation and version information
SycophancyNeutral prompts and independent review in a fresh context
Compounding errorCheckpoints and automatic verification between steps

Asked, or enforced

Part 3 made a distinction that this section depends on. A line in an instruction file or a skill is a request. The model usually honours it, and it is not a control. A guardrail is something that holds whatever the model decides.

LayerExampleHolds if the model is manipulated
Instruction“Never push to main” in the instruction fileNo
Permission setting in the harnessThe push command is on a deny list, or needs approvalYes, within the harness
HookA script that inspects every command and blocks a push to mainYes, within the harness
SandboxThe agent runs in a container with no credentials for the remoteYes
The system itselfThe branch is protected on the server, and the agent’s token cannot write to itYes, whatever runs the agent

Work from the bottom of that table upwards. The strongest control is the one furthest from the model. Use instructions to make good behaviour likely, and the lower layers to make bad outcomes impossible.

Defending against prompt injection

Start from the assumption that an injection will sometimes succeed, and limit what it can achieve. The routes in have multiplied through this module: a web page, a tool result or a tool’s own description from part 7, a retrieved passage from part 8, text inside an image from part 6, and an installed skill from part 4.

  • Least privilege. Read-only by default. Narrowly scoped tokens. No production credentials in a development agent.
  • Break the dangerous combination. Part 3 of the language models module names it: private data, untrusted content and an outbound channel, all in one agent. Remove at least one. An agent that browses the web gets no secrets. An agent with repository access gets a network allow-list.
  • Approval gates. A person confirms consequential actions: sending messages, merging, deploying, deleting, spending money.
  • Isolate untrusted content. Process it in a separate model call that has no tools, and pass on only a structured result, such as a category or extracted fields. A sub-agent with no tools and no private context is a convenient way to do it.
  • Sandboxing. Run agent actions in a container or virtual machine with no access to the host file system and restricted network access.
  • Keep secrets out of the context. Credentials are injected by the tool layer at execution time. The model never sees them, so it cannot be tricked into revealing them.

Classifiers that detect injection attempts are a useful extra layer. They are not sufficient alone. They are models too, and an attacker who can rephrase until the agent obeys can rephrase until the classifier passes it.

Treat output as untrusted input

Model output that flows into another system needs the same handling as user input. Validate it against a schema. Escape it before rendering in a browser. Use parameterised queries, and never build SQL from it directly. Never pass it to eval. Before installing a package the model suggested, check that it exists and is the one you meant, because part 3 of the language models module describes how invented package names are exploited.

Handling variability

When validation fails, retry and include the validation error in the retry, so that the model can correct itself. Provide a fallback for repeated failure. Give actions idempotency keys, so that a retried step does not send two emails or create two tickets. Put limits on steps, time and spend for every agent run.

Approval without fatigue

If every action needs approval, people stop reading and click yes. Reserve approval for actions that are irreversible or high-impact, and let low-risk ones proceed. Make each request informative: show the exact command, or the actual diff, not a summary of intent.

Remember too that every approval gate is a point where the system runs at human speed. Place each one deliberately, and ask what would need to be true, in sandboxing, permissions or automated checks, for it to be removed.

Observability

Log prompts, tool calls, outputs, approvals, token counts and cost, grouped into one trace per task. Include what was retrieved and which skills were loaded, since part 8 showed that most diagnosis starts there. You need this to debug failures, which cannot otherwise be reproduced. You need it for audit, which matters in any regulated business. And you need it as the raw material for the evals in the next section.

2. System-level evals

In plain terms

An eval is a test suite for an AI feature: a set of realistic inputs and a way of scoring the outputs. It tells you whether the thing works, and whether a prompt change or a new model made it better or worse. It lets you argue from evidence. Without one, every change is a guess and every vendor claim is unverifiable. It is the most important practice in this module and the one most often skipped.

Who should read it: everyone can read the main text. Skip the deep dive.

Why ordinary tests are not enough

A unit test asserts one exact result. An AI feature has many acceptable outputs, varies between runs, and fails partially. So the question changes from “does it pass?” to “how often does it succeed across a representative set?”

Build the dataset

  • Start with 20 to 50 cases. A small, real set beats a large synthetic one.
  • Draw them from real usage or your backlog. This is the same asset part 2 of the language models module recommends for comparing models.
  • Include easy, typical, hard and adversarial cases, including an injection attempt if the feature reads untrusted content, and ugly scans if it reads documents.
  • Turn every production failure into a new case. The set becomes your regression suite.
  • Hold some cases back, and do not tune prompts against them. Otherwise you fit the prompt to the test, the same fault part 2 of the language models module criticises in public benchmarks.

Three kinds of grader

GraderExamplesUse
CodeExact match, schema valid, tests pass, quoted text really appears in the sourceFirst choice. Cheap, fast and reliable
ModelAn LLM scores the output against a written rubricFor open-ended quality. Must be calibrated against human labels
HumanAn engineer reviews a sampleThe reference standard. Use it to calibrate the others and for spot checks

For a coding agent, a case is a repository state plus a task. The grader checks that hidden tests pass, that the lint is clean, that the test files were not tampered with and that the diff touches only what it should. Record cost, time and number of steps alongside success.

Evaluate the parts as well as the whole

This module’s pieces can each be measured alone, and a failure is much easier to find that way.

PieceQuestionMeasure
Instruction file or promptDid the change help?Success rate on the set, before and after
SkillIs it loaded when it should be, and only then? Does it improve the result?Activation rate on prompts that should and should not trigger it. Success with and without the skill
ToolDoes the model pick the right tool and fill in valid arguments?Share of cases with the correct call
RetrievalDid the right passage reach the context?Share of questions whose marked passage is in the top results, as part 8 described
Document readingAre the extracted fields right?Field-by-field accuracy against hand-labelled documents

Some harnesses now ship this for their own extensions. Claude Code, for example, can run a plugin against a set of test prompts several times with and without the plugin loaded, to show what the plugin contributes.

Read the outputs first

Before building any grader, read 50 real outputs and sort the failures into categories by hand. This shows what actually goes wrong, which is rarely what you assumed. Teams that skip this step build elaborate scoring for the wrong problem.

Reading the numbers

Run each case three to five times, because results vary between runs. Track success rate, cost per task, latency and the mix of failure categories.

Be honest about sample size. With 30 cases, a difference of under 10 to 15 percentage points could easily be chance. Small sets are good for catching regressions and large differences. They cannot settle a close contest.

When to run them

  • on every change to a prompt, tool, skill or instruction file, ideally in CI
  • before upgrading a model version
  • when a new model is released, to compare it in an afternoon
  • continuously, on a sample of production traffic

Monitoring in production

Watch the signals users give without being asked: whether suggestions are accepted, edited or regenerated, and whether people give up on a task. Score a sample of live outputs with your graders. Put cost and latency on a dashboard, and alert on the failure rate as you would for any service.

Working is not the same as helping

An eval answers “does it do the task?”. It cannot answer “was the task worth doing this way?”. A feature can pass nine cases in ten and leave the work around it no faster, because the time went to checking its output, or because the step it sped up was never the constraint. That second question needs a figure from before the feature existed: how long the job took, how often it went wrong, what it cost. Take it before launch, because it cannot be recovered afterwards, and people’s sense of how much a tool helps them is reliably generous. Then report the outcome, such as tickets resolved or time to release, and not the usage. Section 7 of part 2 of the AI in the organisation module covers the measures.

3. A design review checklist

In plain terms

These are the questions to ask when someone proposes an AI feature or workflow, whether your own team, a vendor or another department. A proposal that cannot answer them is not ready.

Who should read it: everyone. This section is the module in usable form.

Purpose and fit

  • What does success look like, and how will it be measured?
  • Does this need an agent, or would a fixed workflow with a model call do?
  • What happens today without it, and what is the cost of a wrong output?
  • What is the baseline: how long it takes, how often it goes wrong and what it costs now? Who measured it, before this was built?
  • Where is a person in the loop, and does that make the whole thing run at human pace? What would have to be true to take them out of the routine cases?

Context

  • What does the model need to know, and where does that come from?
  • Which instruction files and skills shape it, who owns them, and are they short?
  • Is the source material accurate, current and free of contradictions?
  • Does retrieval respect the permissions of the person asking?
  • How large is the context per call, including images and tool definitions, and is it laid out for caching?

Tools and permissions

  • Which tools and servers can it reach, and is each one necessary?
  • Which skills, plugins and servers came from outside, and who reviewed them?
  • What is the most damaging thing it could do with the access it has?
  • Does it combine private data, untrusted content and an outbound channel?
  • Which rules are requests to the model, and which are enforced by something else?
  • Which actions need a person’s approval, and will that person see enough to judge?

Failure handling

  • What happens when the output is invalid, wrong or missing?
  • Is output validated before anything downstream uses it?
  • Are there limits on steps, time and spend?

Evidence

  • Is there an eval set drawn from real cases, and what is the current success rate?
  • Which delivery outcome should move, and by how much, against the baseline? When will that be checked?
  • Which model version is pinned, and what is the process for upgrading it?
  • Are prompts, skills and instruction files in version control and reviewed?

Operations

  • What will it cost per task and per month, using the model from part 3 of the language models module?
  • Are prompts, tool calls, retrieved passages and outputs logged and traceable?
  • Who owns it once it is live?

The whiteboard version

Assume the model will sometimes be wrong and sometimes be fooled. Give it the least access that does the job, put the controls in software and not in the prompt, keep a record of everything, and keep a set of real cases that tells you the success rate. Then you can let it run.

Say it two ways

IdeaTechnical versionNon-technical version
GuardrailA control outside the model that bounds the effect of a wrong or manipulated outputA safety rail. We do not count on the AI never slipping. We make sure a slip cannot take it over the edge
SandboxAn isolated execution environment with restricted file and network accessA sealed room for the AI to work in, where mistakes cannot reach anything important
Least privilegeGranting only the minimum permissions and scopes the task requiresThe AI gets the keys to the rooms it needs and no others
The dangerous combinationPrivate data, untrusted content and an outbound channel in one agent enable exfiltration by injectionIf it can read our secrets, read strangers’ text and send messages out, a stranger can ask it to post our secrets. We always remove one of the three
Approval gateA human confirmation required before an irreversible or high-impact actionThe AI does the routine work alone and asks before anything that cannot be undone
EvalA dataset of representative cases with automated graders, run on every changeA test suite for the AI feature. It tells us the success rate, and whether a change helped or hurt
LLM as judgeUsing a model with a rubric to grade open-ended outputs, calibrated against human labelsUsing one AI to mark another’s work, after checking that its marking agrees with ours
TraceThe full record of one task: prompts, retrieved passages, tool calls, outputs and costThe flight recorder. When something goes wrong, we can see exactly what it was told and what it did

Misconceptions to correct

“We have told it not to, so it will not”

True
Current models follow clear instructions well, and a written prohibition prevents most accidents.
Misleading
An instruction is text in the context, competing with every other piece of text there, including text an attacker wrote. Nothing enforces it. Under manipulation, or deep in a long task, it can lose.
What to say
“Instructions make good behaviour likely. For anything that must not happen, we remove the ability: no credentials, no write access, or a check in software that blocks it.”

“A person approves everything, so it is safe”

True
A person reviewing each action can catch what the model gets wrong, and for irreversible actions a person should.
Misleading
People asked to approve everything stop reading within days, and the system then has the appearance of oversight without the substance. It also runs at human speed.
What to say
“We save approvals for the few actions that cannot be undone, and we show the approver the exact command or change. Everything else is made safe by limits and checks, so that nobody needs to watch it.”

“We tried it and it worked”

True
Trying it is the right first step, and a good demonstration shows that the task is possible.
Misleading
A handful of attempts says little about a system whose output varies and whose failures are partial. It says nothing about next month’s model or last week’s prompt change.
What to say
“A demo shows it can work. A set of fifty real cases, run every time we change something, shows how often it does. We decide on the second.”

Core explanations

The ideas from this module that you will explain most often, each in one line for a non-technical listener, with the part that holds the technical version and the detail. The complete two-audience tables remain at the end of each part. The AI in the organisation module and the language models module keep lists of their own.

IdeaSay it like thisPart
The window is a productThe box is the small part. Someone chose the engine, the house rules and what happens to what you typePart 1: AI chat
Outgrowing chatWhen you do it every day, or it has to run without you, the window is no longer the right shapePart 1: AI chat
In-app AIThe AI built into software we already use. Someone else wrote its instructionsPart 2: AI in apps
Vendor studyTheir numbers, from their consultants, about a company that does not existPart 2: AI in apps
Context engineeringWriting a good brief. The AI knows only what we put in front of it, so what we include decides the resultPart 3: Intent, AGENTS.md and other instruction files
Instruction fileThe note we would leave a new colleague on day one, which the AI reads at the start of every sessionPart 3: Intent, AGENTS.md and other instruction files
Asked versus enforcedWe can ask the AI not to do something, and it usually will not. If it must never happen, we lock the door as wellPart 3: Intent, AGENTS.md and other instruction files
AgentAn AI that works through a task step by step, checking the result of each step before the nextPart 5: Agentic work
SkillA procedure manual the AI takes off the shelf only when that job comes upPart 4: Skills, agents and plugins
Sub-agentA helper sent off to do the legwork, who comes back with the answer and not the pile of paperPart 4: Skills, agents and plugins
Multi-modal modelAn AI that can be shown pictures and played recordings. It reads them well and measures them badlyPart 6: Multi-modal models
Tool useThe AI asks our software to do something. Our software decides whether to do it and hands back the resultPart 7: MCP servers and tool use
MCPA standard plug that connects AI tools to our systems, so we build each connection oncePart 7: MCP servers and tool use
RAGIt looks things up in our documents before answering, and shows where the answer came fromPart 8: Retrieval
GuardrailsWe do not count on the AI never slipping. We make sure a slip cannot do much harmPart 9: Guardrails and evals
Least privilegeThe AI gets the keys to the rooms it needs and no othersPart 9: Guardrails and evals
EvalA test suite for the AI feature. It tells us the success rate, and whether a change helped or hurtPart 9: Guardrails and evals
BaselineThe numbers from before we built it. Without them we can show that it works and never that it helpedPart 9: Guardrails and evals

Misconceptions quick reference

All twenty-seven misconceptions from the practical AI module, each with a one-line response. Every one contains some truth, so open by agreeing with that. The full entries, with what is true and what is misleading, are in each part.

ClaimShort responsePart
“The model is the product”We are choosing a product, not just a model. The instructions, the memory and the contract matter as much as the enginePart 1: AI chat
“You have to word it just right”There are no secret phrases. Say what you want, who it is for, and hand it what it needs to knowPart 1: AI chat
“It is only a chat, so nothing leaves the room”The question is not whether the tool is allowed. It is which account, under what termsPart 1: AI chat
“The AI in our tools is safer, because we already use those tools”Being an existing supplier is a good start, not an answer. We ask what each feature can seePart 2: AI in apps
“The vendor’s study shows it pays for itself”Their figure is about a company that does not exist. Ours takes a fortnight to measurePart 2: AI in apps
“It summarised the thread, so I know what happened”Good enough to catch up. Before I act on it, I open the threadPart 2: AI in apps
“Prompt engineering is about finding the magic words”There are no secret phrases. If the output is poor, the brief was missing something, and we fix the briefPart 3: Intent, AGENTS.md and other instruction files
“A longer instruction file makes a better agent”We write down what it cannot find out for itself, and keep it to a page or twoPart 3: Intent, AGENTS.md and other instruction files
“It is in the instruction file, so the agent will not do it”The file shapes what it does. For anything that must never happen, we also make it impossiblePart 3: Intent, AGENTS.md and other instruction files
“An agent is just a smarter model”The model is the engine. Whether the vehicle gets anywhere depends on what we build around itPart 5: Agentic work
“Agents, skills and plugins are three names for the same thing”A skill is something it knows, an agent is somewhere it sends work, and a plugin is the box they ship inPart 4: Skills, agents and plugins
“If the agent ignored my skill, the skill is broken”It picks by the description. If ours is not chosen, we rewrite it in the words someone would usePart 4: Skills, agents and plugins
“A skill is only text, so it is safe to install”For an agent, instructions are code. We install skills the way we install softwarePart 4: Skills, agents and plugins
“More agents will get it done faster”We add a helper when one agent runs out of room or time. One well-briefed agent with good tests beats a committeePart 5: Agentic work
“Cloud agents mean we ship more”They change what we can attempt. Whether it reaches a customer depends on review, so we widen that firstPart 5: Agentic work
“It sees the image the way I do”It is an excellent reader and a poor measurer. We check counts, positions and exact figuresPart 6: Multi-modal models
“It watched the whole video”It saw a frame a second and heard the soundtrack. For anything quick or small, we send the short clipPart 6: Multi-modal models
“Multi-modal means it can make images and video too”Understanding a picture and making one are different tools. We check what each model takes in and gives outPart 6: Multi-modal models
“The AI is running commands on our systems”The AI proposes and our software disposes. What it can reach is a setting we choosePart 7: MCP servers and tool use
“MCP is a standard, so an MCP server is safe to connect”A standard plug does not make every appliance safe. We approve servers like any supplierPart 7: MCP servers and tool use
“The more tools we connect, the more capable the agent”A short list of clear tools works better and is easier to defend than everything we ownPart 7: MCP servers and tool use
“RAG eliminates hallucination”It works from our documents and shows its sources. We still check that the sources say what it claimsPart 8: Retrieval
“We need to fine-tune it on our data”We give it our information at question time. That takes days, and we can see exactly what it was toldPart 8: Retrieval
“Context windows are huge now, so retrieval is obsolete”If it fits comfortably, we paste it in. Most of what we know does not fit, so the AI looks things upPart 8: Retrieval
“We have told it not to, so it will not”Instructions make good behaviour likely. For what must not happen, we remove the abilityPart 9: Guardrails and evals
“A person approves everything, so it is safe”We save approvals for what cannot be undone, and make the rest safe with limits and checksPart 9: Guardrails and evals
“We tried it and it worked”A demo shows it can work. Fifty real cases, run on every change, show how often it doesPart 9: Guardrails and evals

Glossary

Terms introduced in this part, in plain language and in alphabetical order. Earlier terms are defined in parts 1 to 8 and in the glossaries of the language models module.

Approval gate
A point where a person must confirm before an action goes ahead
Baseline
The figures for a piece of work before the AI feature existed: how long it took, how often it went wrong, what it cost
Classifier
A model that sorts input into categories, such as “injection attempt” or “safe”
Eval
A set of test cases with a scoring method, used to measure how well an AI feature works
Grader
The part of an eval that decides whether an output is acceptable: code, a model or a person
Guardrail
A control outside the model that limits the harm a wrong or manipulated output can do
Held-out set
Test cases kept aside and never used while tuning, to give an honest measure
Idempotency key
An identifier that ensures a repeated request takes effect only once
Least privilege
Giving a system only the permissions it needs for its task
LLM as judge
Using a model to grade the outputs of a model
Observability
Logging and tracing that lets you see what a system did and why
Position bias
A judge model’s tendency to favour an answer because of where it appears
Regression suite
Tests kept to make sure that problems fixed once do not return
Rubric
Written criteria that a grader scores against
Sandbox
An isolated place for an agent to run, with restricted access to files and the network
Trace
The full record of one task: every prompt, tool call, output and cost

Sources

This part sets out established practice from the drafter’s general knowledge, and most of it carries over from the part of the language models module that this module replaced. Two details come from documentation read in September 2026.