Inside a Multimodal Model

Lesson 2 of 3 in Multimodal Models.

Strip the marketing away and the dominant documented design is almost disappointingly simple: a Vision encoder turns the image into token-like embeddings, and those embeddings join the same stream the text tokens live in. The transformer downstream does not maintain a separate “image pathway” — it sees a sequence of positions, some of which happened to come from a Tokenizer and some from an encoder looking at pixels.

The concrete documented example on our map is Microsoft’s Phi-4-Reasoning-Vision-15B, whose card describes a SigLIP-2 vision encoder in a mid-fusion architecture (source) — “fusion” being the term of art for where and how the image-derived embeddings meet the language model’s stream. Be careful with the generalization, though: families wire this differently, cards vary enormously in how much architecture they disclose, and some (like DeepSeek’s V4.1-Flash card, which describes a “Causal Encoder-Decoder (CED)” design — source) depart from the common recipe in documented ways. The encoder-into-the-stream picture is the default hypothesis you bring to a new card, not a law.

A vertical stack diagram read bottom to top. Bottom layer: inputs — an image plus a text prompt. Next: the text goes through a tokenizer while the image is split into patches. Next: a vision encoder converts the patches into embeddings. Next: a projection maps those embeddings into the same space as text token embeddings. Next, emphasized: the joint token stream where image-derived and text tokens sit side by side as positions. Top: transformer layers attending over all positions and emitting text tokens.

The common documented pattern, bottom to top: an image is split into patches, a vision encoder embeds them, a projection aligns them with the model’s embedding space, and they join the text tokens in one stream that the transformer processes uniformly. Layout is illustrative — the documented instance is the Phi-4-reasoning-vision-15B card’s SigLIP-2 encoder with mid-fusion; other families differ. (illustrative — source: Documented encoder example: Phi-4-reasoning-vision-15B model card)

Here is the payoff of that picture: everything this site taught you about the stream still applies. Self-attention operates over positions without caring where their embeddings came from — a text token can attend to an image patch’s embedding exactly as it attends to another word. The Context window is still the budget of positions. Decoding is still one text token at a time (unless the card documents another output modality). The mental model you built for a text-only Large language model (LLM) transfers almost intact; what changed is who is allowed to write into the stream.

That transfer is why map-reading beats memorization here. When a new multimodal model lands, you do not need a new theory — you need three answers from its documentation: what encoders feed the stream (modalities in), what decoders read from it (modalities out), and what each non-text input costs in stream positions. The next lesson chases that third answer to the meter.

Why images eat your context budget

The context window is denominated in positions, and the encoder path manufactures positions. An image is cut into patches; each patch (or pooled group of patches) becomes one embedding; each embedding occupies one slot of the window. That is the entire mechanism — there is no separate “image budget”.

The cleanest documented number on our map is Gemma 3’s: the card states images are “normalized to 896 x 896 resolution and encoded to 256 tokens each” (source). Read what that sentence encodes: normalization means the file’s resolution is not what matters — the model sees a fixed-size view, so a screenshot and a poster cost the same 256 tokens on this model. Multiply by the number of images in a prompt and you have the context spend before a single word of text is counted.

Two hedges. First, the per-image cost is model-specific: other cards document different encodings, tiling schemes for large images, or per-resolution costs — the number 256 is Gemma 3’s fact, not a constant of nature. Second, cards may budget directions separately: the same Gemma 3 card documents “Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size” and, separately, “Total output context of 8192 tokens” — input and output windows are distinct documented facts, exactly like input and output modalities. The downstream consequences are mechanical: more image tokens mean a longer prefill (attention’s cost grows steeply with sequence length), a larger KV cache, and less room for your actual text.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.