Multimodal

Models Beginner Also called: multimodal AI, vision

Definition

A model that can take in more than one kind of input: text, images, audio, video, and documents. Older models handled text only. With a multimodal model you can drop in a screenshot, a photo of a receipt, or a PDF and ask about it directly. Images and files use context window space just like text.

In plain English

Text-only models could read. Multimodal models can also look and listen. Take a photo of the error message, the whiteboard, the receipt, and ask.¹

In the real world

Dropping a screenshot into Claude or ChatGPT and asking “what’s wrong here.”²
Photographing a menu in another language.
Uploading a PDF instead of copying the text out of it.

When to use it

Anything that’s easier to show than describe. Layouts, charts, error screens, forms.
Documents where copying the text loses the structure.

When not to use it

Exact numbers from a dense table. Export the data instead.
Anything confidential you wouldn’t paste as text. A photo is data too.

Good, bad, ugly

Good: “look at this” replaces three paragraphs of description.
Bad: it’s confident about what it sees, including when it misreads.
Ugly: screenshots of dashboards, contracts, and faces go into chat tools all day with no one thinking about where they end up.

Key facts

Inputs: Text, images, audio, video, documents
In practice: Drag a photo or PDF into the chat and ask about it
Still counts: Images and files use context window space like text does

Common questions

Can I just upload a photo and ask?

Yes, in most current chat apps. Anthropic's docs describe uploading or dragging an image into the Claude chat window like any file.

Does it read handwriting and screenshots?

Usually well enough to be useful, not well enough to skip checking. Numbers and small text are where it slips.

Is multimodal the same as image generation?

No. Multimodal is about what goes in. Image generation is about what comes out. Some tools do both.

References

What is multimodal AI?IBM · 2024-07-15View source
VisionAnthropic, Claude Platform DocsView source