Multimodal AI

Definition

Multimodal AI is a system that takes more than one kind of input — pixels, text, audio, video — and reasons over them together. The step everybody hand-waves is the first one, so here it is concretely: an image is not shown to the language model as an image. It is chopped into a fixed grid of small squares, each square is turned into one vector, and those vectors are inserted into the same sequence the text tokens occupy. After that the model has no way to tell which of its inputs began life as pixels.

The Qwen2-VL team state the conversion outright for their own model: an image of 224x224 pixels, encoded by a vision transformer with a patch size of 14, "will be compressed to 66 tokens before entering LLM" (Wang et al., 2024). Sixteen patches across by sixteen down is 256 patches; an adjacent-2x2 merge folds those into 64; two marker tokens bracket the result. Sixty-six.

That number is the whole subject in miniature. A picture the size of a postage stamp costs about as much of your context window as 55 words of English. A 12-megapixel phone photo, handed to the same encoder at full resolution, costs more than a 12,000-word document. Vision is not a free extra input channel — it is a very expensive way to spend the budget you were already spending on text.

How It Works

Step one: the image becomes a sequence

The mechanism comes from the Vision Transformer. Dosovitskiy et al. describe it in one sentence: "we reshape the image x ∈ R^(H×W×C) into a sequence of flattened 2D patches ... where N = HW/P² is the resulting number of patches, which also serves as the effective input sequence length for the Transformer" (Dosovitskiy et al., 2020). Each flattened patch is multiplied by a trainable matrix to produce a vector of the model's width D, a learnable position embedding is added so the model knows where in the grid each patch came from, and the resulting sequence goes into an ordinary transformer stack.

Everything about the cost follows from that formula, because P is squared in the denominator. The paper spells out the consequence: "the Transformer's sequence length is inversely proportional to the square of the patch size, thus models with smaller patch size are computationally more expensive." Halving the patch side quadruples the sequence.

Those are not hypotheticals. ViT's own training runs were all done at a resolution of 224, and the naming convention encodes the patch choice — "ViT-L/16 means the 'Large' variant with 16×16 input patch size". CLIP's best model is a ViT-L/14 given one additional epoch of pre-training at 336 pixels, which is the 576-patch row.

Step two: patch vectors are pushed into the language model's vocabulary space

A vision encoder's output vectors are not word embeddings; they live in a space of the encoder's own choosing, with the encoder's own width. Something has to translate. In LLaVA that something is a single matrix: "we apply a trainable projection matrix W to convert Z_v into language embedding tokens H_v, which have the same dimensionality as the word embedding space in the language model" (Liu et al., 2023). The authors describe training that matrix, with the encoder and the LLM both frozen, as "training a compatible visual tokenizer for the frozen LLM."

That phrase is the honest description of what a multimodal model is. Tokenization turns text into integers that index rows of an embedding table; a projector turns pixels into vectors of exactly the same shape as those rows, skipping the integer stage entirely. There is no image tensor sitting off to one side that a special vision module consults. There is one sequence, and attention runs over all of it, so a text token asking "what colour is the sign?" can attend directly to whichever handful of patch vectors happen to cover the sign.

Step three: the two spaces have to have been aligned in advance

A projection matrix only works if there is something to project onto. That alignment is what contrastive pretraining buys. CLIP was trained on 400 million image–text pairs scraped from the web, on the single task of predicting which caption belongs to which image. Each batch forms every image–text similarity score, and cross-entropy is applied along both axes, so a matched pair is pulled together while the mismatched pairs in the same batch are pushed apart. The difficulty of that task scales with batch size, which is why CLIP used "a very large minibatch size of 32,768" and a learnable temperature initialised at the equivalent of 0.07 (Radford et al., 2021); the loss function page works through why the shape of that gradient matters.

What comes out is a single embedding space holding both pictures and sentences. That is the property everything downstream depends on: you can classify an image by embedding the sentence "a photo of a tabby cat" and measuring the angle between two vectors, with no cat training data anywhere in the pipeline.

And the bill arrives in tokens

Because patches become tokens, image inputs are charged against exactly the same budget as text, and they are charged by pixel area. Running the patch arithmetic across realistic image sizes, with Qwen2-VL's patch size of 14 and its 2x2 merge:

Two things fall out of that table. Resolution is quadratic, so an image five times larger in each dimension is a 25x bill — which is why high-resolution modes that tile a picture into several crops and encode each one multiply the count rather than nudging it. And video is the same arithmetic with a frame rate attached: forty frames at 1,523 tokens each is 60,920 tokens for forty seconds of footage, before anybody has typed a question.

Types

The oldest and most widely used typology in this field classifies fusion by where in the pipeline the modalities meet. Baltrušaitis et al. present it as the model-agnostic family, split "into early (i.e., feature-based), late (i.e., decision-based) and hybrid fusion" (Baltrušaitis et al., 2017). The distinction is not academic: it decides what happens when one modality is missing, and how much cross-modal detail the model can ever use.

Early fusion

The modalities meet immediately after feature extraction, before any reasoning happens — in the simplest case by concatenating their feature vectors. Everything described above is early fusion: LLaVA's projected patch vectors are concatenated into the text sequence and the transformer sees one undifferentiated stream. The survey names the advantage plainly — it "can learn to exploit the correlation and interactions between low level features of each modality," and it "only requires the training of a single model."

The cost is rigidity. Every modality must be present, aligned and time-synchronised at the same moment, because there is only one model and it expects one concatenated input. And the input length is the sum of all modalities, which is precisely the token bill in the table above.

Late fusion

The modalities never meet inside the network at all. Each one runs its own model to a complete prediction, and only the predictions are combined — by averaging, voting, weighting by channel noise, or a small learned combiner. A lip-reading system that scores an audio recogniser and a visual recogniser separately and reconciles the two answers is doing this.

What it buys is independence: "it allows for the use of different models for each modality," makes prediction easy "when one or more of the modalities is missing," and can even be trained without parallel data. What it costs is stated just as flatly in the survey: "late fusion ignores the low level interaction between the modalities." No amount of combining two answers recovers the fact that the speaker's lips formed a b exactly when the audio was ambiguous.

Hybrid fusion

Both at once: an early-fusion model and the individual unimodal predictors, with their outputs combined. It has been used for multimodal speaker identification and multimedia event detection. The cost is the obvious one — you are now training and serving several models instead of one, for a gain that has to be measured rather than assumed.

A fourth arrangement sits outside this three-way split, in what the same survey calls model-based fusion, where the joining is a designed part of the architecture rather than a wrapper around it. Flamingo is the clearest example: a Perceiver Resampler compresses any number of image or video features into a fixed 64 visual tokens, and freshly initialised gated cross-attention layers are inserted between the frozen language model's existing layers, where "the keys and values in these layers are obtained from the vision features while the queries are derived from the language inputs" (Alayrac et al., 2022). The modalities meet inside the stack, repeatedly, at a fixed and resolution-independent token cost — the structural alternative to paying for every patch in the main sequence.

Real-World Applications

Be My AI, inside Be My Eyes. The clearest deployment of image-plus-text in a product where the answer matters. OpenAI's GPT-4V system card records that Be My Eyes and OpenAI began collaborating in March 2023, piloted with "nearly 200 blind and low vision beta testers", and that by September 2023 the beta group "had grown to 16,000 blind and low vision users requesting a daily average of 25,000 descriptions". The same document records the failure that comes with it: testers found "the model can make basic errors, sometimes with misleading matter-of-fact confidence."

Stable Diffusion's text encoder is a CLIP encoder. The Stable Diffusion v1 model card describes it as a latent diffusion model "that uses a fixed, pretrained text encoder (CLIP ViT-L/14)", whose output is fed into the UNet through cross-attention. This is the alignment property being cashed in the other direction: because CLIP's text tower already points at the same region of space as its image tower, a frozen text encoder is enough to steer image generation without training a text model at all.

Cross-modal search and zero-shot classification. Embedding a photo library and a query sentence with the same model turns "find pictures of a red bicycle" into a nearest-neighbour lookup, and turns classification into comparing an image vector against a handful of sentence vectors. No labelled examples of the class are needed, which is what made CLIP a component rather than a model.

Open-weight vision-language models in ordinary products. LLaVA showed that a frozen CLIP encoder plus one projection matrix plus a frozen LLM is enough to get a working visual assistant, and Qwen's VL series pushed the same recipe to variable-resolution inputs. Screenshot understanding in a coding agent, receipt extraction in an expenses tool, and diagram question-answering all sit on this stack — as do the consumer assistants, ChatGPT and Gemini among them, where accepting a photo is now a default rather than a feature.

Key Concepts

Modality is a channel of input with its own statistics — pixels, characters, waveform samples. Fusion is the act of combining them; alignment is the prior act of arranging two spaces so that fusion means anything.

The projector (or connector) is the small piece that does the work. LLaVA's is a single matrix; LLaVA-1.5 replaced it with a two-layer MLP and reported that the extra representational power alone improved multimodal capability; Qwen2-VL uses an MLP that also merges 2x2 blocks of patches to cut the token count fourfold. It is the cheapest part of the system to train and the part that determines whether the encoder and the language model can talk at all.

A visual token is a token. It occupies a slot in the sequence, it is attended over quadratically, it lands in the KV cache, and it is billed. Nothing about it is special except its provenance.

Resolution is a token-budget decision, not an image-quality decision. Choosing 336 pixels over 224 with a patch size of 14 is choosing 576 patches over 256 — a 2.25x increase in everything downstream. Providers expose this as a quality setting; underneath it is arithmetic.

Challenges

Image tokens crowd out the text you actually wanted the model to read. This is the practical failure teams hit first. Paste six 1920x1080 screenshots into a request and, by the table above, you have spent roughly 15,500 tokens before the instructions. In an agent loop that keeps its history, every one of those screenshots is re-sent on every subsequent turn — the statelessness cost described under context window, except the unit is now a picture. Agents that browse or use a computer hit this within a few dozen steps, which is why screenshot downscaling and dropping stale frames from history are standard.

Small text disappears in the resize, and no prompt recovers it. A fixed-resolution encoder at 224x224 with 14-pixel patches has exactly 16 patches across the image, whatever the original was. On a 1024-pixel-wide screenshot each patch therefore covers a 64x64 block of the original — a whole word or two averaged into a single vector. Sixteen-pixel body text ends up under four pixels tall after the resize. The model is not failing to read; the characters were destroyed before it ran. Higher-resolution encoders, tiling and variable-resolution schemes all exist for this one reason, and all of them are paid for in tokens.

The model can answer without looking, and training rewards it for doing so. This is the most useful thing on this page to know, because it is invisible in a demo. Language alone predicts a great many answers: in the original VQA dataset, "tennis" was correct for 41% of questions beginning "What sport is", and for questions starting "Do you see a...", "blindly answering 'yes' without reading the rest of the question or looking at the associated image results in a VQA accuracy of 87%" (Goyal et al., 2017). On the rebalanced VQA v2 benchmark the authors built to counter this, a language-only model that never receives the image scored 44.26% against 62.27% for the full multimodal model — a system with the picture entirely removed got most of the way there. When the same models were tested on balanced data instead of the biased data they were trained on, MCB's overall accuracy fell from 60.36 to 54.22, and its yes/no accuracy from 81.20 to 70.40. Treat any multimodal evaluation without a text-only baseline as unreported.

Confident wrong descriptions are worse here than in text. A hallucinated citation can be checked; a hallucinated object in a photograph can only be checked by someone who can see the photograph, which in the accessibility case is precisely the person who cannot. The Be My AI beta explicitly surfaced this, and it is why such products are built around asking rather than asserting.

Interleaving and ordering are underspecified. With several images in one request, the model's ability to keep track of which question refers to which picture depends on markers in the sequence and on how the training data was formatted. Numbering images explicitly in the prompt is not a superstition; it is compensating for a genuinely ambiguous input format.

Future Trends

Variable token budgets per image are replacing the fixed square. ViT's original design fixed the input resolution, so every image was squashed to the same grid regardless of what it was. Qwen2-VL removed the absolute position embeddings in favour of 2D rotary embeddings specifically so it can "process images of any resolution, dynamically converting them into a variable number of visual tokens." The direction of travel is that a plain photograph is cheap, a dense document page is expensive, and the model decides rather than the preprocessing script.

Token compression is where the engineering effort is going. A 2x2 merge is a 4x saving that Qwen2-VL ships by default; Flamingo's resampler pins the cost at 64 tokens regardless of input size. Every one of these is a bet about how much of a patch grid is redundant, and video — where the naive count is frames times per-frame tokens — is where the bet pays or fails most visibly.

Native multimodal pretraining versus bolting an encoder on. The LLaVA recipe trains a projector between two frozen models that were pretrained separately. Training on interleaved images and text from the start produces different internal representations, at much greater cost, and the open question is whether the resulting quality gap justifies giving up the ability to swap either component.

Audio and video are following vision through the same pipeline, not around it. A spectrogram is patched exactly like an image; a video is patched in space and time. Whatever solves the token-cost problem for pictures will decide whether long-form video generation and video understanding become routine or stay expensive.

Code Example

The whole cost model, in arithmetic you can check. This uses Qwen2-VL's published configuration — patch size 14, a 2x2 merge into one token, and two marker tokens — so the first row must come out at the 66 tokens the paper states.

PATCH = 14   # pixels per side of one patch (CLIP ViT-L/14, and Qwen2-VL's encoder)
MERGE = 2    # Qwen2-VL merges each 2x2 block of patches into a single token
EXTRA = 2    # the <|vision_start|> and <|vision_end|> markers

def visual_tokens(width, height, patch=PATCH, merge=MERGE, extra=EXTRA):
    cols = (width // patch) - (width // patch) % merge
    rows = (height // patch) - (height // patch) % merge
    patches = cols * rows
    return patches, patches // (merge * merge) + extra

print(f"{'image':>12}{'patches':>10}{'tokens':>9}{'~words of text':>16}")
for w, h in [(224, 224), (336, 336), (1092, 1092), (1920, 1080), (3024, 4032)]:
    patches, tokens = visual_tokens(w, h)
    print(f"{f'{w}x{h}':>12}{patches:>10,}{tokens:>9,}{round(tokens / 1.21):>16,}")

# A 40-frame video clip at 1 frame per second, same encoder.
_, per_frame = visual_tokens(1092, 1092)
print(f"\n40 frames x {per_frame:,} tokens = {40 * per_frame:,} tokens for 40 seconds of video")

Its output:

       image   patches   tokens  ~words of text
     224x224       256       66              55
     336x336       576      146             121
   1092x1092     6,084    1,523           1,259
   1920x1080    10,336    2,586           2,137
   3024x4032    62,208   15,554          12,855

40 frames x 1,523 tokens = 60,920 tokens for 40 seconds of video

The word-equivalents use 1.21 tokens per English word, the ratio measured on this site's own prose under o200k_base on the tokenization page. Change MERGE to 1 to see what the same images cost without the compression step, and PATCH to 16 to see the ViT-B/16 grid instead. Every provider's exact accounting differs; the shape — quadratic in resolution, linear in frames — does not.