Search Results Annotations at Google

Google search results go way beyond the regular 10 blue links. Ever wondered how the enrichments (or snippets) are displayed in search engine results?

In this post, I will cover some of what I have learned reading the Google patent titled “Search Results Annotations” by Denise Ho, Grzegorz Glowaty, Reed Taylor, Tom Murphy and Juro Gottweiss.

This post does not cover rich results that is shown when a website adds specific structured data to a web page. We will instead focus on annotations that are not related to “regular” search results.

Subscribe to my Newsletter

What are Annotations?

Annotations are additional data that relate to a search result, but that is not typically included in a regular search result.

In a nutshell, it is any data that is not a link, a title, a description, a favicon, or a rich snippet.

Rich snippets are things like star ratings and prices.

Some of the annotations are query dependent and some not. This means that some annotations are stored offline and others are computed at query time.

The annotations may be displayed in a carousel, a swipe-and- grow interface, an
expandable row, etc.

Below are examples of annotations.

Pros and cons

Information extracted by Google from pages with multiple reviews available. Likely the most mentioned n-grams. Not necessarily as exact match with text on page.

UPDATE: Google now also support pros and cons through structured data, though it will keep extracting the data without the markup

Editorial Reviews

Reviews are query-based annotations and can be ranked according to the relevance of the review to the query.

List Includes

Top Pick

Top pick is a potential enhancement that we may see coming up as a feature enrichment in SERP, but no example to share so far.

Additional Annotation Examples

There are other types that are mentioned in the patent for which I have no examples to show:

  • Mention annotations: Elements from articles that mention the item (e.g. News article about a product). Can take the form of a snippet, a summary.
  • Top ranked: Item was ranked in a list of “Top X” items in a category (e.g. Top Beach Vacations Spots, Top Gifts on Father’s Day)
  • Version change: For replaced or updated products with newer version available (iphone 10 vs Iphone X)
  • New release: Newly released products. New products may have more weight.
  • Product attributes: Specifications such as the size, the colour, etc.
  • Description: related to a search query when description is too long or not available.
  • Multimedia: Element such as a video, image or audio file describing an item.
  • Prior query annotation

Possible Annotations Not Mentioned in Patent

Image thumbnails

Shown when the query is an image seeking query with enough high-quality images available.

Table description

Coming when the table html tag is relevant to the query.

What is the Patent About?

The “Search Results Annotations” provides insights on how annotations are ranked and displayed within search results to provide enrichment that goes beyond the 10 blue links.

The patent was filed in 2021 and has a pending status.

Patent Discussed About

  • Annotations
  • SERP enrichment
  • Indexing
  • Crawling
  • Ranking
  • Query processing

Highlights From Search Results Annotations

  • Annotations can compete against each other depending on viewport size
  • Annotations can appear alongside rich snippets
  • For certain query type, image thumbnails are annotations, for others not (e.g. Shopping marketplace). This means that image thumbnail may never be shown if more relevant annotation has a higher priority
  • Annotations show the item id, annotation type, payload (full or light), the score, the kind of item (doc, entity,…)
  • Annotations can be computed at query time (query dependant) or be pre-computed and stored in an index
  • If a highest ranked annotation does not meet a certain threshold, no annotations will be shown.
  • Annotations may be shown in different shapes and sizes, with user interactions needed, or not.

How Are Annotations Chosen?

Annotations can come from one or more internal or external sources and the annotations can be scored using the framework described in the patent described in the current article.

A machine-learned scoring algorithm may rank all annotations for an item (e.g. web document).

Then, depending on the size of the screen of a given device (viewport), the system may display lightweight annotations or full annotations.

For example, a space-constrained mobile device may show the lightweight annotation and any devices larger than a laptop may be exposed to full annotations.

Example annotation ranking pipeline from the “search results annotations” patent

Annotation Pipeline

Here is the process by which Google may annotate search results:

  1. The query engine processes the query and determines what indexes to search from (web, images, hotels, marketplace, etc.).
  2. It ranks documents of a primary repository based on the query.
  3. It determines if it need to look at offline annotations, or need to process the annotation at query-time.
  4. Then, one or more annotators produce the annotations for each item using secondary repositories.
  5. The annotator(s) may rank annotations against each other using machine learning.
  6. The models can be trained using human raters where raters score the usefulness of the annotation.
  7. The annotator engine aggregates and ranks scored annotations produced by the annotator
  8. The query system then defines the number of items, the size and the way the annotations will be shown to the user.
Generating dynamic annotations for a search result page

What Does an Annotation Look Like?

Annotations are appending data to a search results.

Here an example of how an annotation may look like.

Example of a normalized annotation format

The table below shows the data that is stored with the annotation.

Item_IDIdentifies the item in the first repository the annotation applies to
TypeAnnotator type (Editorial, pros/cons, …)
PayloadIncludes the data to be included in an annotation. (lightweight/full, instructions to display items). Dependent on the annotation type.
ScoreScore of annotation. Could also be annotation features
Annotation identifierWhat kind of item it identifies (document, product, entity,…)

Search Engine Environments Where Auxiliary Information may be Taken From

The patent mentions a few kinds of search engines where the current patent may gather auxiliary information to fill-up annotations.

  • Web search engine
  • Media search engine (images)
  • Marketplace search engine
  • Travel Search Engine
  • New Home Search Engine
  • Entity Search Engine

Additional Annotation Information on Images

An additional patent named “Clustering Queries for Image Search” lists annotation information that is stored for images, it is unclear if and how this would be used in the context of the “Search Results Annotations” patent.

Image annotation information that is listed in the patent as being included is:

  • titles,
  • descriptions,
  • ratings,
  • Sources for the images,
  • geographic locations at which the images were captured,
  • Information provided by web page administrators, image providers or third parties.
  • Advertising information


Patent termDefinition
AnnotationsAuxiliary data about search results that are not part of the fundamental items (link, title and description) usually included in a search result
ItemsElements of different formats that can be contains in an index. (document, product, book, application, hotel, restaurant,…)
Query dependant annotationAnnotation that has to be processed at query time since it uses parts of the query to generate annotations
Offline annotationPre-computed annotations that are stored and assigned prior to a query
AnnotatorSoftware that generates the annotation payload of an item for the search engine
Item repositoryItems from a specific search engine corpus (web, images, marketplace)
Primary RepositoryRepository defined as the primary corpus by the query engine based on the query
Secondary RepositoriesAdditional repositories used to provide annotations for the ranked items of the primary repository

Google Search Infrastructure Involved

The “Search Results Annotations” patent mentions these elements from the Google Search Infrastructure:

  • Query System
    • Query Engine
    • Annotator Engine
    • Indexing Engine

Patent Details

NameSearch Results Annotations
AssigneeGoogle LLC
Appl No.17/303,206
Inventor(s)Grzegorz Glowaty and al.

Other Similar Patents


This patent was very interesting to understand what is going on when Google modifies the appearance of search results without the website owners specifically requesting those changes.

5/5 - (1 vote)