Modalities In, Modalities Out

Lesson 1 of 3 in Multimodal Models.

“Multimodal” is the most overloaded word on any vendor page, and the way to defuse it is a single question asked twice: what goes in, and what comes out? Input modalities and output modalities are independent axes. A model that reads images fluently may only ever emit text. A family that markets image generation may route it to a separate model entirely. Until you have read both axes off the Model card, you know nothing.

The clearest documented illustration of the two axes is Meta’s Llama 3.2 Vision card, which states the collection ships “in 11B and 90B sizes (text + images in / text out)” — its capability table lists input modalities as “Text + Image” and output modalities as “Text” (source). Images in; text out; nothing implied beyond that. Contrast Qwen2.5-Omni-7B, whose card says it is designed to “perceive diverse modalities, including text, images, audio, and video” while “generating text and natural speech responses in a streaming manner” (source) — four modalities in, and two out. Same word, “multimodal”; very different machines.

Key terms: Multimodal, Modality, Vision encoder, Model card, Context window

Across the landscape, the documented pattern repeats with variations. Meta describes Llama 4 models as “natively multimodal” (source). Qwen’s current flagship card claims “Native support for image and video understanding, from STEM diagrams and documents to hour-scale videos” (source) — note that understanding is input-side language. Mistral publishes a dedicated Voxtral audio family alongside its text models (source). On the API side, OpenAI’s docs state that “All latest OpenAI models support text and image input, text output, multilingual capabilities, and vision” — while image generation, realtime speech, and transcription appear as separately listed models on the same page (source). Anthropic’s docs make the same two-axis statement for Claude: “All current models support text and image input, text output, multilingual capabilities, vision, and tool use” (source). Google’s Gemini docs likewise list image, video, music, and live-voice models as their own entries in the catalog (source).

Two navigation lessons hide in that survey. First, text remains the spine: in every documented case above, text is an input and text is an output — other modalities attach to a text-centric core rather than replacing it. Second, output modalities beyond text tend to live in specialized models, even inside families whose main line reads images natively. When a vendor page says “the family does X”, your next click is the specific model’s documentation.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.