The paper applies a pure Transformer encoder to image classification by splitting the image into fixed-size patches and treating each patch as a token in a sequence. It is the architecture behind Google's modern visual-search stack and the visual encoder of the multimodal models that now ground images, queries, and pages together.
Patent Overview
- Inventor
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby
- Assignee
- Google LLC
- Filed
- 2020-10-22 (arXiv 2010.11929)
- Granted
- ICLR 2021; patent family active at Google
The Challenge
The Challenge
Convolutional networks had dominated vision for nearly a decade, but their inductive bias for locality and translation invariance was both a help and a ceiling. Convolutions cannot easily attend to long-range relationships in an image, they require deep stacks to widen the receptive field, and they were a different primitive from the attention layers that now powered language. The team needed a way to apply the same Transformer that revolutionized text directly to pixels, without smuggling convolution back in through the side door.
- Convolutional Bias Caps The Ceiling — Per layer, convolutions hard-code locality and translation invariance. The bias helps at small data scale but caps accuracy when data and compute grow large.
- Long-Range Vision Was Expensive — Per scene, relating a logo in one corner to a product in another required deep convolutional stacks. The model could not attend across the image cheaply.
- Vision And Language Used Different Stacks — Per modality, language ran on Transformers while vision ran on CNNs. The split blocked unified multimodal architectures.
- Pixels Are Not A Natural Token Sequence — Per image, a 224x224 pixel grid yields 50,176 positions. Direct token-per-pixel attention is quadratic and infeasible at that length.
- Pre-Training Recipes Did Not Transfer — Per architecture, the masked and contrastive pre-training that powered language Transformers had no clean analog in convolutional vision. Vision needed a primitive that could absorb the same scaling laws.
Innovation
How The System Works
The system splits the image into a grid of non-overlapping patches, flattens each patch into a vector, linearly projects it to a token embedding, adds a learnable position embedding, and feeds the resulting sequence into a standard Transformer encoder. A learnable classification token aggregates the image representation and feeds a classification head. No convolutions are used.
- Split Image Into Fixed-Size Patches — Per image, the input is divided into a grid of non-overlapping patches, typically 16x16 pixels. A 224x224 image becomes a sequence of 196 patches.
- Flatten And Linearly Project Each Patch — Per patch, the pixels are flattened into a vector and projected through a single learned linear layer into a token embedding of the model's hidden size.
- Prepend A Learnable Classification Token — Per sequence, a special CLS token is prepended. Its final representation will be used as the image-level embedding for classification.
- Add Learnable Position Embeddings — Per position, a learned position embedding is added to each patch token so the encoder knows where the patch sat in the original grid.
- Process With A Standard Transformer Encoder — Per layer, multi-head self-attention and feed-forward sublayers operate on the patch sequence. The architecture is the same encoder used for language.
- Read Out The CLS Token — Per image, the final representation of the CLS token is taken as the image embedding and passed to a small classification head.
- Pre-Train Large, Fine-Tune Per Task — Per model, the encoder is pre-trained on very large image datasets such as JFT-300M or ImageNet-21k, then fine-tuned on downstream tasks like ImageNet-1k or CIFAR.
An Image Is A Sequence Of Patches
The load-bearing idea is that an image can be tokenized just like a sentence. Each 16x16 patch becomes a word in a visual vocabulary, position embeddings encode where it sat in the grid, and the same Transformer that processes language processes the picture. Vision becomes a sequence-modeling problem.
Patches Are Visual Tokens
Per image, a small grid of patches gives the Transformer a sequence short enough to be tractable and rich enough to carry the image's meaning. The model learns which patches matter for which class by attending across the whole sequence at every layer.
- Pure Transformer, No Convolution — Per layer, only self-attention and feed-forward sublayers are used. No convolutional inductive bias is injected anywhere in the stack.
- Scale Beats Inductive Bias — Per scaling regime, ViT trails CNNs on small data but matches and surpasses them once pre-training data reaches hundreds of millions of images.
- Unified With Language Stacks — Per architecture, vision now shares the same primitive as language. Multimodal grounding becomes a natural extension rather than a bolt-on.
Technical Foundation
Technical Foundation
The paper specifies the patch tokenizer, the linear projection, the CLS token, the learnable position embeddings, the standard Transformer encoder, and the pre-train then fine-tune recipe.
- Patch Tokenizer — Per image, the input is reshaped into a sequence of flattened 16x16 (or other size) patches.
- Linear Patch Embedding — Per patch, a single learned linear layer projects the flattened pixels into the model's hidden dimension.
- Learnable CLS Token — Per sequence, an extra token is prepended whose final state acts as the image representation.
- Learnable Position Embeddings — Per token, a learned 1D position embedding is added. Spatial structure is recovered from data, not hard-coded.
- Standard Transformer Encoder — Per layer, multi-head self-attention plus a feed-forward network, with layer norm and residual connections, identical to the language Transformer.
- Pre-Train And Fine-Tune Recipe — Per deployment, large-scale supervised pre-training on JFT-300M or ImageNet-21k is followed by fine-tuning on the target task.
The Process
The Process
Training and inference follow a standard supervised pipeline, but the data scale and the patch tokenizer are the defining choices.
- Choose Patch Size And Resolution — Per model, patch size (typically 16x16) and input resolution are chosen. Smaller patches mean longer sequences and finer detail.
- Build The Patch Embedding Layer — Per architecture, a single linear projection maps flattened patches into the encoder's hidden size.
- Pre-Train On Web-Scale Imagery — Per pre-training run, the model is trained on hundreds of millions of labeled images. Scale is what unlocks ViT's accuracy.
- Fine-Tune On Target Tasks — Per task, the pre-trained encoder is fine-tuned on ImageNet-1k, CIFAR, or other downstream benchmarks with a fresh classification head.
- Interpolate Position Embeddings For Resolution Change — Per fine-tune, if the input resolution changes, the learned position embeddings are interpolated to match the new patch grid.
- Evaluate Against CNN Baselines — Per benchmark, ViT is compared against the best convolutional baselines to confirm the scaling crossover holds.
- Reuse The Encoder Across Modalities — Per multimodal system, the ViT encoder is reused as the visual backbone for image-text models such as CLIP, ALIGN, PaLI, and Gemini Vision.
Quality Control
Quality Control
A pure Transformer on pixels has no inductive bias to fall back on, so the training recipe carries safeguards that keep the model learning the right structure.
- Pre-Training Data Scale — Per model size, pre-training data must be large enough for attention to learn spatial structure from data. JFT-300M and ImageNet-21k are the proven regimes.
- Patch Size Discipline — Per task, patch size balances sequence length against detail. Smaller patches read finer features but cost more compute.
- Position Embedding Validation — Per training run, position embeddings are inspected to confirm they capture 2D structure, not just sequence order.
- Fine-Tune Resolution Handling — Per fine-tune, position embeddings are interpolated when resolution changes so the patch grid stays geometrically consistent.
- Benchmark Across Tasks — Per backbone, the same encoder is validated across classification, transfer, and downstream multimodal tasks to confirm generality.
Real-World Application
ViT is the visual encoder behind Google's modern image stack. Google Lens, Image Search ranking, the image pack on web results, product detection in Shopping, and the visual half of multimodal systems including CLIP-style retrieval and Gemini Vision all sit on ViT and its descendants.
- 16x16 Patch Size — Each image is split into a grid of 16x16 patches as tokens.
- Pure attention Encoder — No convolutions. Self-attention and feed-forward layers only.
- Web-scale Pre-training — Hundreds of millions of images give ViT its accuracy edge.
Why Patches Beat Pixels
Per image, treating individual pixels as tokens is infeasible because attention is quadratic in sequence length. Patches give the encoder a sequence short enough to process and long enough to retain meaning. The patch grid becomes the visual vocabulary.
Why ViT Became The Visual Half Of Multimodal
Per multimodal system, ViT exports a token sequence in the same shape that language Transformers consume. CLIP, ALIGN, PaLI, and Gemini Vision use ViT-style encoders so image tokens and text tokens can share an attention space. Image-text grounding follows from architecture, not from a hand-built bridge.
<\/section>What This Means for SEO
What This Means for SEO
Google's image understanding now runs on ViT-style encoders, not CNN feature extractors. That changes how Image Search, Lens, image-pack ranking, and multimodal grounding read your visuals. Image composition, distinctive elements, and the relationship between images and their surrounding text all matter more than they did under the convolutional stack.
- Visual Search And Lens Run On ViT — Google Lens, Image Search, and image-pack ranking are powered by Vision Transformers and their descendants. Image understanding is patch-level attention, not CNN feature extraction. The model reads the whole composition holistically and weighs every patch against every other patch.
- Composition And Layout Matter For Ranking — Treating an image as a sequence means patch ordering, layout, and spatial relationships are all captured. Where the product sits, how the logo relates to the packaging, what surrounds the subject, all enter the encoder. Composed imagery beats centered-subject stock photos.
- Alt Text And Captions Are Co-Trained Signal — ViT-based multimodal models are pre-trained on web-scale image and text pairs. The encoder has learned associations between visual patterns and language directly from the open web. Alt text, captions, filenames, and the paragraph nearest the image teach the model what your image depicts.
- Distinctive Visual Brand Assets Are Read — Patch attention means small image elements such as logos, product labels, packaging shapes, and distinctive details get tokenized and weighted independently. Visible brand assets in imagery contribute to entity association. Brand-bearing visuals carry more signal than generic equivalents.
- Generic Stock Photography Reads As Low-Signal — Large-scale pre-training favors common visual patterns. Generic stock photographs collapse into the same dense region of representation space and carry weak distinguishing signal. Semantically distinct, original imagery competes far more strongly for image-pack and Lens placement.
- Image And Text Are Scored Together — ViT underpins multimodal grounding in CLIP, PaLI, and Gemini Vision. Your image and your text are scored in the same embedding space, not independently. Mismatched alt text, captions, or surrounding context costs you signal because the alignment between the two modalities is itself a ranking input.
- Long-Tail And Zero-Shot Image Classes Still Get Recognized — ViT generalizes to image classes it never explicitly trained on through the learned representation space. Niche product types, specialized equipment, or unusual subjects can still be recognized via their visual structure and their text grounding. Long-tail imagery is not invisible to the model.