A multimodal model is an AI model that can work with more than one kind of input, such as text, images and audio, and sometimes video too. Show one a photo of a dog, play it a bark and ask 'What is this?', and it can combine all three to answer: 'Dog.'
One model, many kinds of input
Each kind of data is called a modality. Most AI models used to handle just one:
- Text models read and write words.
- Vision models read pictures: they classify photos or spot objects.
- Speech models hear sounds: they turn speech into text, or text into speech.
Each one works well in its own world, but knows nothing about the others. A text model can't look at your screenshot, and a vision model can't read your question about it.
A multimodal model pulls them together. It takes in several modalities at once, connects them, and reasons across them. That matters because real problems are rarely just text: a bug report comes with a screenshot, a lecture is speech plus slides, a medical case is scans plus notes.
How it works: encoders and a shared space
A model can only do maths on numbers, so the first job is turning every kind of input into numbers it can compare.
Step 1: encoders turn data into numbers
Each modality has its own encoder: a part of the model that turns raw data into lists of numbers called embeddings.
- An image encoder typically cuts the picture into small square patches and turns each patch into an embedding. A 224 by 224 pixel image cut into 16 by 16 patches gives 14 × 14 = 196 patches, so the model 'sees' it as 196 embeddings.
Wondering how this was made? How we make these