Uses a BERT bidirectional transformer to read a question and a candidate passage together, then a conditional random field to extract the exact answer span, replacing keyword-match retrieval with contextual language understanding for natural-language queries.
Patent Overview
- Filed
- 2020-09
- Granted
- 2021-05
- Application Number
- CN 112115238A
The Challenge
The Challenge
Question answering on web text was held back by keyword retrieval. The system could find passages mentioning the right terms but could not understand what was actually being asked, especially when prepositions, word order, or syntax carried the meaning rather than the keywords.
- Keyword Match Misses Question Semantics — A user asking 'who can drive without a license' wants an exception, not a license-holder. Keyword retrieval returns pages about drivers and licenses, missing the negation entirely. The system needs to read the question, not just match its words.
- Prepositions And Word Order Carry Meaning — 'Flight from Boston to LA' and 'flight from LA to Boston' contain the same keywords in different orders. Bag-of-words retrieval cannot distinguish them. Question answering requires understanding syntactic role.
- Answer Extraction Needs Span-Level Precision — Even when the right passage is retrieved, the answer is usually a phrase inside the passage, not the whole thing. The system needs to identify the exact span (a date, a name, a number) rather than returning a paragraph.
- Pronouns And Coreference Break Pattern Match — Documents use pronouns and indirect references. 'He founded the company in 1998' is meaningful only with the antecedent of 'he' in scope. Surface-pattern extraction misses these constructions entirely.
- Long-Range Context Matters — A passage's meaning often depends on language several sentences away. Without a model that handles long-range dependencies, the system cannot resolve which entity an answer refers to when multiple are mentioned in the same passage.
Innovation
How The System Works
The patent encodes question and candidate passage together through BERT, producing contextual representations that capture word meaning conditioned on the full context. A conditional random field over the passage tokens then identifies the answer span by labeling each token as start, inside, or outside the answer.
- Concatenate Question And Passage — The question text and the candidate passage text are concatenated with special separator tokens. The combined sequence is fed to BERT as a single input, so attention layers can relate words across both.
- Encode With Bidirectional Transformer — BERT processes the sequence in both directions simultaneously, producing a contextual vector for every token. Each token's representation now depends on the entire question and passage, not just neighboring words.
- Attend Across Question And Passage — The self-attention layers let passage tokens attend to question tokens and vice versa. A token in the passage learns whether the question is asking about it, and what kind of answer the question expects.
- Predict Token Labels With A CRF — A conditional random field reads the BERT representations and assigns each passage token a label: B (begin answer), I (inside answer), O (outside answer). The CRF enforces global consistency so labels form a valid sequence.
- Extract The Answer Span — The B-I sequence identifies one or more answer spans. The span with highest CRF confidence is returned as the answer. For 'who founded Google', the span is 'Larry Page and Sergey Brin'.
- Score Confidence For Display Decision — Each predicted answer carries a confidence score. If confidence is below a threshold, the system returns the passage without highlighting an answer span, or suppresses the featured answer entirely.
- Aggregate Across Multiple Passages — The same question is asked against multiple candidate passages. Answers are aggregated, with consensus across passages boosting confidence and disagreement triggering disambiguation.
Contextual Embedding Plus Sequence Labeling
The patent's load-bearing combination: BERT for contextual understanding, CRF for structured span output. Either alone is weaker. Together they handle both the language understanding and the precise extraction the answer task requires.
Read Together, Label Together
Encoding question and passage in a single attention window lets the model see the relationship directly. Labeling spans with a CRF enforces that the extracted answer is a contiguous, well-formed phrase. The two halves are designed for each other.
- Bidirectional Attention — BERT's bidirectional encoding means every token's representation reflects the full context, including the question. Words can have different vectors depending on what question they are paired with.
- Joint Question-Passage Encoding — Question and passage flow through the same transformer in the same sequence. Cross-attention is not bolted on, it is the default. The model sees them as parts of one task rather than two separate inputs.
- CRF For Valid Spans — A naive token classifier could label 'B' here, 'O' there, 'I' somewhere disconnected. The CRF rules out invalid label sequences and produces well-formed answer spans, raising precision substantially.
Technical Foundation
Technical Foundation
The patent specifies the model architecture, the training procedure, and the inference pipeline. Each layer was at the technical frontier when the patent was filed.
- BERT Pre-Trained Encoder — The base model is a BERT transformer pre-trained on masked language modeling and next-sentence prediction over a large text corpus. Pre-training gives the model general language understanding before task-specific fine-tuning.
- Fine-Tuning On QA Data — The pre-trained model is fine-tuned on labeled question-answer pairs from SQuAD, Natural Questions, and proprietary Google datasets. Fine-tuning specializes the model for the QA task without forgetting general language knowledge.
- Conditional Random Field Layer — A linear-chain CRF sits on top of the BERT output, modeling label transitions between adjacent tokens. The CRF parameters are learned jointly with the final BERT layer during fine-tuning.
- Self-Attention Mechanism — The transformer's self-attention computes scaled dot-product attention across all token pairs in the sequence. This is what allows long-range and cross-segment dependencies to flow through the model.
- Tokenization Strategy — Text is tokenized into wordpiece subwords so the model handles out-of-vocabulary terms gracefully. Question and passage share the same vocabulary, and special tokens delimit them.
- Inference Optimization — Production deployment uses optimizations: smaller distilled models for cheap queries, model caching, batching across queries, GPU/TPU inference. The patent describes the inference path at production scale.
The Process
The Process
Production BERT-QA runs as a downstream stage after candidate-passage retrieval. The retrieval step is fast and approximate; BERT-QA is the precise re-ranker and answer extractor.
- Retrieve Candidate Passages — An upstream retrieval system (text-match plus other signals) fetches top-K candidate passages for the query. Typical K is 5 to 20 passages.
- Format Question-Passage Pairs — For each candidate, build the [CLS] question [SEP] passage [SEP] sequence. Truncate the passage if the combined length exceeds the model's input limit (typically 512 tokens).
- Run BERT Inference — Feed each pair through the BERT-CRF model. The output is per-token labels and per-token confidence scores. This is the compute-heavy step.
- Extract Predicted Spans — Decode the CRF output to identify answer spans (B-I sequences). Each candidate passage produces zero or more spans with their confidence scores.
- Aggregate Across Candidates — Spans across multiple passages are compared. Identical or near-identical spans reinforce each other; divergent spans reduce overall confidence. The aggregator outputs a final answer plus confidence.
- Decide Display Treatment — If confidence exceeds the threshold, the answer is rendered as a featured snippet with the span highlighted in the source passage. Otherwise the system falls back to standard result links.
- Log For Continuous Training — Each answered query and its outcome (click, dwell, follow-up query) is logged. The signal feeds future model retraining so the system gets better at the kinds of questions users actually ask.
Quality Control
Quality Control
BERT-QA can hallucinate, extract wrong spans, or surface answers from low-authority passages. The patent specifies defenses against each failure mode.
- Confidence Threshold For Featured Display — Featured answers are reserved for spans with high confidence. Below the threshold, the system returns search results without a featured answer. False featured answers cost user trust, so the threshold is set conservatively.
- Source Authority Filtering — Candidate passages from low-authority sources are downweighted or excluded before BERT-QA runs. The system avoids extracting answers from spam, satire, or speculation pages even if the model would extract them correctly.
- Cross-Passage Consensus Check — When multiple passages give different answers, the system either picks the consensus or suppresses the featured answer. Disagreement is a signal that the question is contentious or the system is uncertain.
- Question Type Calibration — The model is calibrated separately for factoid questions, definition questions, and complex multi-part questions. The display logic differs per type, with factoid answers shown most aggressively and complex answers most conservatively.
- Continuous Evaluation — A holdout set of labeled questions is evaluated on every model version. Regressions in accuracy or coverage trigger investigation and rollback. Production models must beat the previous version on the eval set.
Real-World Application
BERT-based QA shipped to Google Search in October 2019 (announced by Pandu Nayak), affecting approximately one in ten queries at launch. Its descendants now power featured snippets, Search Generative Experience, and the question-answering layer of Google's voice products.
- 10% Of Search Queries Affected At Launch — Pandu Nayak's October 2019 announcement said BERT would affect about 10 percent of search queries in English. The percentage grew as the model rolled out to more languages and query types.
- 70+ Languages Supported — BERT was extended to more than 70 languages within a few years of the initial launch. The multilingual variants share the same architecture with language-specific fine-tuning.
- Featured Snippet Quality Lift — Internal evaluations showed substantial accuracy gains on natural-language and conversational queries. Featured snippets became markedly less prone to misleading extraction errors.
Question-Form Content Becomes A Ranking Lever
Once the engine could understand questions, content phrased as question-and-answer became disproportionately extractable. SEO practice around H2 questions plus first-sentence answers traces directly to how this model reads pages and decides what to surface as a featured answer.
From Featured Snippets To Generative Answers
BERT-QA's extraction pipeline is the architectural ancestor of Search Generative Experience and AI Overviews. The progression from extracting spans, to summarizing passages, to generating multi-source answers is a continuous evolution from this patent's core ideas.
<\/section>What This Means for SEO
What This Means for SEO
When the engine reads a question with BERT and walks a knowledge graph to ground its answer, your content has to be structured both for natural-language match and for entity grounding.
- Answer Passages Need Entity Anchors — BERT understands the question linguistically, but answers are validated against a knowledge graph. Wrap key facts in entity-rich language with explicit subject-verb-object structure so the model can map them onto graph relationships.
- Conversational Phrasing Wins Long-Tail Questions — The system handles question-form queries natively. Write H2 and H3 headings that mirror real question phrasing, and answer the question in the first sentence of the section that follows.
- Structured Data Is A Trust Shortcut — A page that asserts the same entity relationships in JSON-LD as it does in prose gives the model two signals that agree. Two agreeing signals are how a passage becomes the chosen answer.