LLM engineering · September 9, 2026
Multimodal LLM Integration: Costs and Failure Modes
Sending an image to a model takes about four lines. What breaks afterwards is token economics, silent misreads, and an injection surface most teams do not know they opened.
By Shihab Shahriar Antor · Updated 2026-09-09
Adding vision to an application is a small diff. You base64 an image, put it in the message content array, and it works on the first try. That experience is misleading, because everything expensive about multimodal shows up later and none of it announces itself.
Images are priced in tokens, and the count moves fast
Providers convert an image into tokens before the model sees it, usually by splitting it into tiles and encoding each one. The exact arithmetic differs per vendor and changes, but the shape is consistent everywhere: cost scales with pixel area, so doubling each dimension roughly quadruples what you pay.
The practical consequence is that a handful of full-resolution screenshots can consume more context than a long document, and it happens without any obvious signal in your code.
| Decision | Effect | What to do |
|---|---|---|
| Resolution sent | The dominant term. Uploading a 4K screenshot when the model will downscale it anyway pays for pixels that are discarded. | Downscale client-side to the largest dimension the provider actually uses. Read the current numbers from the vendor rather than guessing. |
| Detail or fidelity setting | Low-detail modes encode the image at a fixed small cost. The gap between low and high is frequently an order of magnitude. | Default to the cheap mode and escalate only for tasks that provably need it. Most classification and layout questions do not. |
| Number of images per request | Linear, and it compounds with conversation history if images stay in context across turns. | Drop images from history once they have been described. Keep the model's own text summary instead of the pixels. |
| Cropping | Sending the relevant region rather than the whole page cuts cost and usually improves accuracy at the same time. | If you know where the answer is, crop to it. This is the rare optimisation with no tradeoff. |
| Prompt caching | A large static image reused across many requests can often be cached, which changes the economics of document workflows substantially. | Put stable images early in the prompt so they fall inside the cacheable prefix. |
Per-provider token formulas change without notice. Measure your own usage from response metadata rather than computing it from a blog post, including this one.
Where vision models quietly get it wrong
Multimodal failure is rarely an error response. It is a fluent, plausible answer that is wrong, which makes it far more dangerous than a timeout. The failure categories are well defined and worth testing for explicitly.
- 01
Small text in dense visuals
Chart axis labels, table footnotes, legal fine print. Accuracy falls off sharply as text gets small relative to image size, and the model does not report low confidence. It reports a number.
- 02
Counting and spatial precision
How many items, which one is left of which, what value does this needle point at. These remain weak across current models. If a count matters, get it from something other than a vision model.
- 03
Multi-image confusion
Given several images the model loses track of which is which and blends attributes between them. Label each one explicitly in the text and refer to it by that label rather than by position.
- 04
Reading order in complex layouts
Multi-column pages, sidebars and footnotes get linearised in an order the model chooses. Extracted text can be correct in content and wrong in sequence, which silently corrupts anything downstream that assumes order.
- 05
Hallucinated structure
Asked to extract a table, a model will produce a well-formed table even when the region contains no table. Schema-constrained output makes the result parseable, which is not the same as making it true.
Documents: render, extract, or both
PDF handling is where most multimodal work actually happens, and there are three approaches with different economics.
Text extraction is nearly free and loses layout, so it fails on anything where position carries meaning, which includes most forms, invoices and financial statements. Page rendering to images preserves layout and costs image tokens per page, which gets expensive fast on long documents. Sending both, extracted text plus the rendered page, is the most accurate and the most expensive.
The pattern that works is usually tiered: extract text first, and escalate to rendering only for pages where extraction produced something suspicious, such as very little text on a page that should have plenty. Escalation logic is cheap and cuts the bill sharply on documents that are mostly ordinary prose.
Audio and video are text pipelines wearing a costume, until they are not
For most applications, transcribing audio and feeding the text to a model is cheaper, faster and easier to debug than native audio input, and it is the right default. Native audio earns its cost only when something outside the words matters: tone, hesitation, overlapping speakers, a sound that is not speech. If a transcript would have answered the question, use the transcript.
Video is frames plus audio. The frame sampling rate is the entire cost decision, and the right rate depends on whether you are answering questions about what is in the video, where one frame every few seconds is plenty, or about motion, where it is not.
What to build before shipping any of it
An eval set of real inputs, including the ugly ones. Photographs at an angle, scans with artefacts, screenshots at odd aspect ratios, documents in the languages your users actually have. Multimodal accuracy is far more input-dependent than text accuracy, so a demo that works on clean inputs predicts very little about production.
Score extraction against known-correct values rather than by eye, and keep every input that produced a wrong answer as a permanent fixture. The general approach is the same one described in testing LLM applications without fooling yourself, with one addition: for vision, a rising score on a clean eval set is especially misleading, because the failures live at the messy end of the distribution.
Questions
- How much do images cost in an LLM API call?
- It depends on resolution, because providers tile the image and charge tokens per tile. Cost scales roughly with pixel area, so doubling each dimension roughly quadruples the price. Most providers also offer a low-fidelity mode at a fixed small cost, and the gap between that and full fidelity is often an order of magnitude. Read the token counts from your own response metadata rather than estimating.
- Should I send a PDF as text or as images?
- Text extraction when layout does not carry meaning, since it is nearly free. Page images when position matters, which covers forms, invoices, statements and anything with columns or checkboxes. The efficient pattern is to extract text first and escalate to rendering only for pages where the extraction looks wrong, such as a page returning almost no text when it should return plenty.
- Can images be used for prompt injection?
- Yes, and it is one of the least audited attack surfaces in production AI systems. A vision model reads text rendered inside an image and does not reliably separate it from your instructions. Small, low-contrast or corner-placed text can carry instructions past every text-level input filter, because at filtering time the payload is pixels. Any pipeline that accepts user images and has tools attached needs to treat image content as untrusted.
- Why does the model misread numbers in my charts?
- Small text relative to image size is the weakest area of current vision models, and axis labels and data annotations are usually the smallest text on the page. Crop to the region, increase effective resolution for that crop, and where the number is important, obtain it from the underlying data rather than from a picture of it.
- Is native audio input better than transcribing first?
- Only when something beyond the words matters. Transcription then text is cheaper, faster, easier to debug and easier to evaluate, so it is the correct default. Native audio earns its cost for tone, hesitation, speaker overlap, and non-speech sounds. If a transcript would have answered the question, the transcript is the better engineering choice.