Domain-level ranking signals that resist spam manipulation by reading patterns across the whole domain rather than per-page, so a single high-quality page on a spam domain cannot pull the domain up and a thin page on a quality domain inherits the domain's authority.
Patent Overview
- Inventor
- Marc Najork, others
- Assignee
- Microsoft Corporation
- Filed
- 2005-09-20
- Granted
- 2007-03-22 (published application)
- Application Number
- US 11/231,138
The Challenge
The Challenge
Page-level spam detection misses domain-level patterns: a thousand-page spam farm can hide one well-crafted decoy page that fools page-level evaluators. Domain-level aggregation reveals the pattern that page-level analysis cannot.
- Page-Level Detection Misses Domain Patterns — Spam sites can craft individual pages to evade per-page detectors. Looking at the domain as a whole reveals patterns no single page exposes.
- Decoy Pages Fool Per-Page Evaluators — A spam site can include a few high-quality decoy pages to pass per-page checks. Domain-level aggregation sees through this evasion.
- Domain Signals Are Harder To Game — Manipulating per-page signals is cheap. Manipulating the average quality across thousands of pages is expensive. Domain-level signals are structurally more resistant.
- Quality Domains Carry Quality Forward — A new page on a high-quality domain inherits the domain's authority as a prior. The page does not have to prove itself from zero.
- Domain Aggregation Must Be Robust — Aggregation methods (mean, median, percentile) each have manipulation profiles. The system must pick aggregation that resists the manipulation patterns spam sites use.
Innovation
How The System Works
The system computes per-page quality signals across the domain, aggregates them robustly to produce a domain-level quality score, applies the domain score as a prior for new pages on the domain, and combines per-page and per-domain signals in the final ranking.
- Compute Per-Page Quality Signals — Per page, standard quality signals: content evaluation, spam detection, structural quality, behavioral signals. These feed both per-page ranking and domain-level aggregation.
- Aggregate To Domain Level — Across the domain, aggregate per-page signals using robust statistics (median, trimmed mean, percentile). Aggregation resists manipulation by a few outlier pages.
- Detect Domain-Level Spam Patterns — Specific patterns flag spam at the domain level: high page count plus low quality variance, thin content across many pages, link patterns suggesting farm structure.
- Compute Domain Quality Score — Combine aggregate quality signals plus spam-pattern indicators into a single domain quality score. Score is bounded and calibrated against engagement outcomes.
- Apply As Page-Level Prior — New or under-evaluated pages on the domain inherit the domain score as a prior. Strong domain pulls the page up; weak domain pulls down.
- Combine With Per-Page Signals — Final ranking combines domain prior with per-page signals. Strong per-page signals can override domain prior; weak ones get the domain treatment.
- Refresh As Domain Evolves — Domain quality changes over time. Periodic refresh keeps the prior aligned with current domain state.
Domain As Quality Unit
The patent's load-bearing idea is to make the domain a first-class unit of quality evaluation. Per-page signals contribute to domain assessment; the domain in turn shapes per-page ranking. The two layers reinforce each other.
Manipulation Resistance Comes From Scale
Manipulating a few pages is cheap; manipulating thousands is structurally expensive. Domain-level aggregation makes manipulation cost-prohibitive in a way per-page analysis cannot.
- Robust Aggregation — Per-page signals aggregate via robust statistics. Outlier pages cannot dominate domain assessment.
- Pattern Detection — Domain-level spam patterns (thin content at scale, farm-like link structures) flag domains the page-level detectors miss.
- Page-Level Prior — Domain quality becomes a per-page prior. Under-evaluated pages inherit the domain's standing.
Technical Foundation
Technical Foundation
The patent specifies the per-page signal extractors, the domain aggregator, the spam pattern detectors, the domain score combiner, and the page-level prior application.
- Per-Page Signal Extractors — Standard quality signal extractors: content evaluation, spam detection, structural quality, behavioral signals. Output per-page features for both ranking and aggregation.
- Domain Aggregator — Aggregates per-page features across the domain using robust statistics. Resists manipulation by outlier pages.
- Spam Pattern Detectors — Detect domain-level patterns: high page count plus low quality variance, farm-like link structures, thin content at scale. Patterns flag spam at the domain level.
- Domain Score Combiner — Combines aggregate quality plus spam-pattern indicators into a single domain score. Calibration anchors to engagement outcomes.
- Page-Level Prior Application — Per page, domain score becomes a prior that combines with per-page signals. The combination is bounded to prevent domain quality from completely overriding strong per-page signals.
- Refresh Pipeline — Domain scores refresh as the domain evolves. Refresh cadence balances responsiveness against compute cost.
The Process
The Process
The pipeline runs as a periodic batch over the crawled web. Output is per-domain quality scores that the ranking system reads as priors for individual pages.
- Crawl And Extract Per-Page Signals — Standard crawl produces per-page signals. Quality, spam, content evaluation all run as before.
- Group Pages By Domain — Pages group by their domain. The domain becomes the aggregation unit.
- Aggregate Robustly — Per domain, aggregate per-page features using robust statistics. Output is per-domain feature vector.
- Detect Domain Spam Patterns — Pattern detectors scan for domain-level manipulation signatures. Flagged domains get spam-indicator scores.
- Compute Domain Quality Score — Combine aggregate features and pattern flags into final domain score. Score is bounded and calibrated.
- Publish To Ranker — Domain scores publish to the ranker's feature store. Per page, the ranker reads its domain score as a prior.
- Combine At Query Time — Per query, the ranker combines per-page signals with domain prior. Final score determines ranking.
Quality Control
Quality Control
Bad domain assessment penalizes legitimate sites or misses manipulators. The patent specifies safeguards.
- Robust Aggregation — Median, trimmed mean, and percentile-based aggregation resist outlier influence. No single page can dominate domain assessment.
- Pattern Detector Calibration — Spam pattern detectors are calibrated against confirmed examples. False positives penalize legitimate large sites; false negatives miss manipulation.
- Bounded Prior Influence — Domain prior cannot completely override strong per-page signals. Excellent pages on weak domains still get fair treatment.
- Per-Subdomain Granularity — Large domains with mixed subdomains can score per-subdomain rather than per-root-domain. Granularity prevents over-aggregation.
- Refresh Cadence — Domain scores refresh as quality evolves. Improving domains get credit; declining ones lose it.
Real-World Application
Domain-level quality signals are core to modern web search ranking. The primitives this patent describes underpin domain-authority scoring, site-quality penalties, and the per-domain reputation layer in major search engines.
- Domain-aggregated Signal Unit — Quality signals aggregate at the domain level. Domain is a first-class unit, not just a URL substring.
- Robust Aggregation Statistics — Robust statistics resist outlier manipulation. Spam sites cannot inflate domain scores through a few decoy pages.
- Bounded Prior Influence — Domain prior bounds influence on per-page ranking. Strong per-page signals can still override the prior.
Why Site-Level Quality Compounds Per-Page Ranking
Pages on high-quality domains inherit prior credit. Investment in domain-wide quality (editorial standards, content depth, technical health) compounds visibility for every page on the site, not just the page being optimized.
Why Mass Content Production Hits Domain Caps
Producing thousands of thin pages dilutes domain quality. The aggregator detects the pattern and applies a domain-level penalty that affects every page. Mass production strategies that worked in 2010 fail under modern domain-aware ranking.
<\/section>What This Means for SEO
What This Means for SEO
This patent aggregates per-page quality across a whole domain using robust statistics and applies the domain score as a prior for individual pages, while detecting farm-like patterns. SEO implication: site-wide quality compounds for every page, and mass thin-content production drags the entire domain down.
- Site-Wide Quality Compounds Per Page — A new page on a high-quality domain inherits the domain score as a prior rather than starting from zero. Investing in domain-wide editorial standards lifts visibility for every page, not just the one you are optimizing.
- Mass Thin Content Hits A Domain Cap — Producing thousands of thin pages lowers aggregate domain quality and triggers a domain-level penalty affecting every page. Scaled low-value production that worked years ago now backfires under domain-aware ranking.
- Decoy Pages Do Not Fool Aggregation — Robust statistics like median and trimmed mean resist a few high-quality decoy pages on an otherwise weak domain. You cannot lift a low-quality site by polishing a handful of pages.
- Excellent Pages Survive Weak Domains — The domain prior is bounded and cannot fully override strong per-page signals. A genuinely excellent page on a weaker domain still gets fair treatment, so quality is never wasted.
- Aggregating Across Pages Is Hard To Game — Manipulating a few pages is cheap; raising average quality across thousands is expensive. The domain unit is structurally resistant to the page-level tricks that once worked.
- Subdomain Granularity Can Apply — Large mixed sites may score per-subdomain rather than per-root-domain. Separating high-quality and low-quality content into different subdomains can prevent one section from dragging another down.
- Domain Reputation Is Refreshed — Domain scores refresh as quality evolves, so improving domains earn credit and declining ones lose it. Cleaning up or pruning weak content can recover a domain's standing over time.