Pairs each answer passage with an image drawn from documents that already pair text and images for the same query, so the image surfaced alongside a direct answer is grounded in a real document.
Patent Overview
- Inventor
- Steven D. Baker
- Assignee
- Google LLC
- Filed
- 2017-04-20
- Granted
- 2020-06-23
- Application Number
- US 15/493,002
The Challenge
Pictures Make Answers Better
Text answers are improved by an accompanying image when the question's answer is visually depictable. Pulling the right image from somewhere appropriate, in real time, requires a second retrieval pipeline that respects the original question and the candidate answer. Without constraints, generic image search would return images that match the question topic but not the specific answer being shown.
- Generic Image Search Is Off-Target — Running an image search for the original query produces images that match the question, not images that depict the specific answer being shown. The result is an image that is on-topic for the question but does not illustrate the answer.
- Stock Photo Libraries Lack Context — Pulling from a generic image library produces decorative images that may not match the answer or its source. Stock images have no documentary link to the answer.
- Need A Coupled Text-And-Image Retrieval — The system needs a corpus of documents that contain both text and images so the chosen image is grounded in a real document that addresses the question. Coupling text and image at the document level is the key constraint.
- Reformulation Matters — The original query may not be optimal for image search. A second query, derived from the first, often performs better against image-bearing corpora. The reformulation step is what bridges question to visual answer.
- Image Quality Must Survive Retrieval — Among image candidates from responsive documents, the system still has to pick the best image. Quality signals (resolution, composition, distinctness) gate the final selection beyond responsiveness alone.
Innovation
Reformulate, Search Image-Bearing Resources, Select
The system takes the question query, generates a second query (typically a reformulation focused on the answer shape), searches a corpus of resources that include images alongside textual data, then selects an image from that resource set to accompany the textual answer. The constraint to image-bearing resources is what keeps the image grounded in a documentary source.
- Confirm Question Query — The first query is determined to be a question query suitable for an image-accompanied answer. Not every question warrants an image; the system gates accordingly.
- Generate The Second Query — Generate a second query from the first, typically reformulated to focus on the expected answer type or visual aspect of the question. The reformulation can use synonym substitution, answer-term injection, or template expansion.
- Search The Image-Bearing Corpus — Run the second query against a corpus of resources that contain both images and textual data rendered with the images. Plain text documents do not participate.
- Identify A Responsive Resource Set — From the search, identify the set of resources determined to be responsive to the second query. These are documents that both address the answer and contain images.
- Search Images Within The Set — Run an image search constrained to the responsive resources. Each image in those resources is a candidate.
- Select An Image — Select an image from the candidate set based on additional signals: image quality, distinctness, caption-text alignment, and other ranking inputs.
- Provide With The Answer — Return the textual answer to the user together with the selected image. Source attribution may accompany both the answer and the image.
Constrained Image Retrieval
The patent's contribution is the constraint: image candidates must come from resources that already pair text and images for the same query intent. The constraint ensures the image is documentary, not decorative.
Image From The Document That Answers
An image that comes from a document responsive to the question is more likely to illustrate the answer than an image surfaced by unconstrained image search.
- Two-Stage Query — The original question query and a derived second query. The first establishes intent; the second targets image-bearing resources.
- Constrained Image Pool — Images are drawn only from resources that match the second query. The pool is small and high-precision compared to global image search.
Image search is documentary, not decorative, when constrained to responsive resources.
<\/section>Technical Foundation
Why Constrained Image Search Works
By restricting the image search to resources that already contain a textual response to the question, the system ensures the image came from a document that knows what the answer is about.
- First Query — The original question query that triggered the answer flow. Established by upstream classification as a question.
- Second Query — A query derived from the first, tuned for the expected answer or visual content. The derivation can incorporate answer-type signals from the question classification.
- Image-Bearing Resource — A document or page that contains both images and textual data rendered alongside those images. The pairing is the structural constraint that makes the image documentary.
- Image Candidate Set — The images contained in the responsive resources. The set is searched again to select the best candidate based on image-specific quality signals.
Key Insight: The patent does not propose new image-ranking signals. It proposes a constraint on the candidate pool. By restricting candidates to resources that pair text and image for the same intent, even ordinary image-ranking signals produce documentary outputs. The constraint is the contribution; the ranking is conventional.
<\/section>The Process
End-To-End Image-Answer Pipeline
The image-answer flow runs as an extension of the answer-passage pipeline. After a question query has been identified, the image side runs in parallel and produces an image to surface with the textual answer.
- Question Classification — Upstream classification identifies the query as a question suitable for an image-accompanied answer.
- Second Query Generation — Generate a derived query that targets the image-bearing corpus. The derivation may apply synonym substitution or answer-type injection.
- Responsive Resource Search — Run the second query against the corpus of image-bearing resources. Identify the responsive set.
- Image Extraction — Extract every image from the responsive resources as a candidate. Each candidate carries metadata about its source document, surrounding text, and alt text.
- Image Ranking — Rank candidates using image-specific signals: resolution, composition, alt-text alignment with the question, caption-text alignment.
- Deliver Answer Plus Image — Return the textual answer together with the top-ranked image. Both carry source attribution back to their originating documents.
What This Means for SEO
What This Means for SEO
Images that get pulled into rich answers come from pages that pair them with on-topic text. The implications for image SEO on content pages are direct and shape both content structure and image-metadata strategy.
- Pair Images With Answer-Shaped Text — The image candidates the system considers are bounded by the surrounding textual responsiveness. An image without nearby on-topic text is not an answer candidate even if its alt text matches the query. Place images next to the answer passage.
- Alt Text And Captions Are Retrieval Inputs — Both alt text and visible captions feed into how the constrained image search identifies the image. Both should reflect the answer concept, not just the literal scene depicted.
- Be On The Page That Answers The Question — Images on pages that win the answer-passage scoring inherit eligibility for the image slot. The fastest way to win the image is to win the text answer.
- Use Distinctive Original Images — Constrained image searches tend to select images that are unique to a winning resource. Reused stock photos rarely surface as featured images because they have weak documentary association.
- Resolution And Composition Still Matter — Among the responsive candidates, image-specific quality signals decide the winner. High resolution, clean composition, and a clear subject all help once you have cleared the responsiveness gate.
- Image Filename And Surrounding Text Compound — Filename, alt text, caption, and adjacent body text all contribute to whether the image is recognized as on-topic. Treat them as one coherent metadata stack rather than independent inputs.
- Don't Hide Images Inside JS Lazy-Load Without Markup — Images that require client-side script to even appear in the DOM may be missed by the constrained image search. Server-side rendering or proper noscript fallbacks ensure the image is part of the document the system reads.