Multi-modal Models

011. What multi-modal means

In and out are separate

A mode is a kind of data: text, image, audio, video. A model has a list of modes it accepts and a list of modes it produces, and the lists are rarely the same. The most common arrangement by far is text and images in, text out. Such a model can describe, compare, transcribe and reason about an image. It cannot draw one.

When a chat product appears to do everything, it is usually a language model with tools. It reads your photograph itself, and when you ask for a picture it writes a prompt and calls an image model, as part 7 describes. The join matters when something goes wrong, because the model that understood your request is not the one that drew the result.

The line is blurring. A few models now generate images or speech from the same network that reads them, and the token-based design described in the generative media module is the route by which that is happening. For practical purposes in 2026, check the two lists for the model in front of you.

What it is good for

• reading documents as they are, with their tables, stamps, handwriting and layout
• extracting fields from forms, receipts and invoices into structured data
• explaining a chart, a diagram or an architecture drawing
• checking a user interface: a coding agent that can see the rendered page can fix what it sees
• describing images for accessibility, search or moderation
• transcribing and summarising meetings, with who said what
• operating software that has no API, through screenshots

022. How a model reads a picture

From pixels to tokens

Part 1 of the language models module shows text becoming tokens, and each token becoming a vector. An image takes a parallel path to the same place. It is divided into patches, a small network turns each patch into a vector, and those vectors are placed in the sequence beside the vectors for your words. From there on the transformer treats them alike. Attention lets a word in your question look at a patch of the image in the same way it looks at another word.

This explains the general shape of what these models can do. They are good at what an image is about, how its parts relate, and what the text in it says, because those survive being turned into a few thousand vectors. They are weaker at exact positions, exact counts and fine detail, because those often do not.

What an image costs

The figures differ by vendor, and one documented example shows the pattern. Anthropic’s Claude models use patches of 28 by 28 pixels, and an image costs one token per patch.

tokens = ceil(width / 28) x ceil(height / 28)

1000 x 1000 pixels   ->  36 x 36  =  1,296 tokens
1920 x 1080 pixels   ->  69 x 39  =  2,691 tokens

Each model has a ceiling. On the current high-resolution models it is a long edge of 2,576 pixels and 4,784 tokens, and on older ones 1,568 pixels and 1,568 tokens. A larger image is scaled down to fit before the model sees it. So a 4K screenshot costs no more than the ceiling, and its small text may no longer be legible after the shrink.

At a few dollars per million input tokens, a thousand one-megapixel images cost single-digit dollars. That is cheap for a task and adds up in a loop. An agent that takes a screenshot at every step, and resends the conversation each turn, pays for every earlier screenshot again unless the harness clears them, which is one more reason for the compaction in part 5.

Getting a good reading

• Send the part that matters. Crop to the table, the error message or the component. A crop at full resolution beats a whole page scaled down.
• Resize it yourself. If you shrink the image to the model’s limit, you see exactly what the model will see, and you can check that the small print survived.
• Image first, question after. The advice for long documents applies: material first, then the question.
• Label several images. “Image 1:”, “Image 2:”, so that you and the model can refer to them.
• Keep it upright and sharp. Rotated, blurred and heavily compressed images are documented causes of errors.
• Do not rely on hidden data. The model sees pixels only. It does not receive the file’s metadata, such as the date or the location in a photograph.

033. Documents, charts and screens

How a PDF is read

A PDF may hold real text, pictures of text, or both. Sending only the extracted text loses the tables, the charts and anything scanned. The documented approach for Claude is to do both: each page is converted to an image, the text of the page is extracted, and the model receives the two together. The published estimate is 1,500 to 3,000 tokens a page for the text, plus the image cost from section 2. A hundred-page report can therefore run to several hundred thousand tokens, and dense documents can fill the context before any page limit is reached.

So the questions from part 8 apply. For one report, put it in the context. For ten thousand, retrieve the pages that matter first.

Extraction

Turning documents into data is the workhorse use. Ask for structured output against a schema, as part 7 describes, and apply two rules from it with extra force. Give every field an honest way out, such as null or “unreadable”, because a smudged total will otherwise be given a plausible value. And validate what comes back: totals that should add up, dates that should be dates, identifiers that should match a pattern.

For long runs of clean, printed text, a conventional OCR or PDF parser is cheaper and exact, and the model can work from its output. The model earns its cost where layout carries meaning, where the scan is poor, or where the task needs understanding as well as reading.

Charts and diagrams

Models explain charts well: what is plotted, the trend, the outlier, what the author wants you to conclude. Reading exact values off a chart is less reliable, because a value is a position, and position is what the grid blurs. If the numbers matter, get the data behind the chart. If only the picture exists, ask for values with a stated tolerance and check a sample.

Screens and computer use

A screenshot is an image like any other, and it closes a loop that text cannot. A coding agent that can see the rendered page, through a browser tool, can tell that the button is off the screen or the text is unreadable, which no test of the markup would show.

Computer use goes further: the model is given screenshots and tools to move the pointer, click and type, and it operates an application as a person would. It works, and it is slower and less reliable than an API call, since each step is a screenshot, a decision and an action, and a misjudged coordinate clicks the wrong thing. It is useful for legacy systems that have no API. It also carries the largest injection risk in this module, because everything on the screen is input. Section 5 returns to that.

044. Audio and video

Two ways to handle speech

The older route is a pipeline: a speech-to-text model produces a transcript, and a language model reads it. It is cheap, the transcript can be stored and searched, and any language model will do. It discards everything that is not words: tone, hesitation, overlapping speakers, the alarm going off in the background.

The newer route is a model that takes audio in directly. Google’s Gemini documentation gives a rate of 32 tokens for each second of audio, so a minute is 1,920 tokens and an hour is about 115,000. It lists transcription with timestamps, telling speakers apart, detecting emotion and recognising sounds that are not speech. It also notes that audio is downsampled and mixed down to a single channel, so a recording that kept each speaker on a separate channel loses that separation.

Some models also produce speech directly, which makes spoken conversation possible without a pipeline of three models. Alibaba’s open-weight Qwen3-Omni takes text, images, audio and video, and answers in text or speech. The gain is latency and naturalness. The cost is that everything in this module still applies, to a system that now answers before anyone can review what it said.

Video is frames plus sound

No current model takes in every frame. The same documentation describes sampling one frame a second, at either 66 or 258 tokens a frame depending on the resolution setting, with the audio track alongside. That comes to roughly 100 tokens for each second of video at low resolution and 300 at high.

So a model with a million-token context holds about three hours of video at low resolution or one hour at high, which is what the documentation states. It also means an hour of footage is one of the most expensive single inputs you can send.

One frame a second is enough for a lecture, a meeting or a screen recording. It is not enough for anything fast: a dropped catch, a flicker in an interface, a sleight of hand. Low resolution is enough for who is in the room and not for the text on their slides. Some APIs let you raise the frame rate or trim the clip, and one now offers a mode in which the model navigates the video, looking closely only where it needs to, with a claimed saving of most of the tokens on long footage.

The generative media module explains why generated video is short and drifts. Understanding video has the opposite problem. The clips can be long, and the model’s attention to any one second is thin.

Choosing a route

055. What they miss

The documented limits

Vendors publish these. The list below follows Anthropic’s and is typical.

• Counting. Approximate, and worse with many small objects.
• Spatial reasoning. Positions, coordinates and layouts are approximate. Anything that depends on exact location needs verifying.
• Small, rotated or poor images. Errors rise sharply with low quality, rotation and very small images.
• People. The models will describe a person and will not name one from their face, by policy.
• Synthetic images. A model cannot reliably tell whether an image was generated. Do not use one as a detector. The generative media module covers provenance.
• Specialist images. A general model is not a diagnostic tool for medical scans, and the vendors say so.

Seeing what is expected

The failure that matters most is quieter. A language model predicts what is likely, and that habit does not switch off when the input is an image. Shown a familiar logo with one letter changed, a clock with its hands in an unusual place, or an animal with an extra leg, models often report the familiar version. Shown a form, a model may fill a blurred field with the value such fields usually hold.

This is hallucination, from part 3 of the language models module, with a picture as the prompt. It is dangerous for the same reason: the output is confident and well formed. In document work it means the rare, unusual value, the one that matters most to an auditor, is the one most likely to be normalised away.

Images carry instructions

Part 3 of the language models module describes prompt injection: text the model reads as data is obeyed as an instruction. A model that reads images reads the text in them. Instructions can sit in a screenshot, in a scanned letter, in small pale type on a web page that a person would never notice, or in a frame of a video.

For a model that only answers questions, the damage is a wrong answer. For an agent with tools, and above all for computer use, it is the full risk described in part 9, with a larger attack surface, since filters built for text do not inspect pixels. Treat every image from outside as untrusted content.

Working with the grain

• Ask for what the model is good at: meaning, structure, text, comparison. Verify what it is weak at: counts, positions, exact values.
• Where a number matters, have the model quote the text it read, and check that against the source or a second method.
• Build an eval set of your own documents, including ugly ones, as part 9 describes. Public scores say little about your invoices.
• When the answer is wrong, look at what was sent. Most failures are a detail that did not survive resizing.
• Keep a person on the cases where a misreading is expensive.

A note on choosing

Support for modes is the fastest-changing fact in this module. As of September 2026, in broad terms: all the major hosted model families read images and PDFs; Google’s Gemini models also take audio and video directly; OpenAI’s take images and audio; and the strongest open-weight options are the Qwen3-VL family for images and video, Qwen3-Omni for audio as well, and Google’s Gemma 4, whose smaller models take audio and run on a laptop under an Apache 2.0 licence. Part 2 of the running AI locally module explains how to read those names. Check each model’s card for its two lists before you design around it.

The whiteboard version

Everything becomes tokens. A picture is a grid of patches, a sound is a string of slices, a video is a slideshow with a soundtrack. The model reads them well and cheaply enough to be useful everywhere. It does not measure, it does not count, and it believes whatever is written on the wall.