Google search results go way beyond the regular 10 blue links. Ever wondered how the enrichments (or snippets) are displayed in search engine results?
In this post, I will cover some of what I have learned reading the Google patent titled “Search Results Annotations” by Denise Ho, Grzegorz Glowaty, Reed Taylor, Tom Murphy and Juro Gottweiss.
This post does not cover rich results that is shown when a website adds specific structured data to a web page. We will instead focus on annotations that are not related to “regular” search results.
Navigation
Show
What are Annotations?
Annotations are additional data that relate to a search result, but that is not typically included in a regular search result.
In a nutshell, it is any data that is not a link, a title, a description, a favicon, or a rich snippet.
Rich snippets are things like star ratings and prices.
Some of the annotations are query dependent and some not. This means that some annotations are stored offline and others are computed at query time.
The annotations may be displayed in a carousel, a swipe-and- grow interface, an
expandable row, etc.
Below are examples of annotations.
Pros and cons
Information extracted by Google from pages with multiple reviews available. Likely the most mentioned n-grams. Not necessarily as exact match with text on page.
UPDATE: Google now also support pros and cons through structured data, though it will keep extracting the data without the markup
Editorial Reviews
Reviews are query-based annotations and can be ranked according to the relevance of the review to the query.
List Includes
Top Pick
Top pick is a potential enhancement that we may see coming up as a feature enrichment in SERP, but no example to share so far.
Additional Annotation Examples
There are other types that are mentioned in the patent for which I have no examples to show:
- Mention annotations: Elements from articles that mention the item (e.g. News article about a product). Can take the form of a snippet, a summary.
- Top ranked: Item was ranked in a list of “Top X” items in a category (e.g. Top Beach Vacations Spots, Top Gifts on Father’s Day)
- Version change: For replaced or updated products with newer version available (iphone 10 vs Iphone X)
- New release: Newly released products. New products may have more weight.
- Product attributes: Specifications such as the size, the colour, etc.
- Description: related to a search query when description is too long or not available.
- Multimedia: Element such as a video, image or audio file describing an item.
- Prior query annotation
Possible Annotations Not Mentioned in Patent
Image thumbnails
Shown when the query is an image seeking query with enough high-quality images available.
Table description
Coming when the table html tag is relevant to the query.
What is the Patent About?
The “Search Results Annotations” provides insights on how annotations are ranked and displayed within search results to provide enrichment that goes beyond the 10 blue links.
The patent was filed in 2021 and has a pending status.
Patent Discussed About
- Annotations
- SERP enrichment
- Indexing
- Crawling
- Ranking
- Query processing
Highlights From Search Results Annotations
- Annotations can compete against each other depending on viewport size
- Annotations can appear alongside rich snippets
- For certain query type, image thumbnails are annotations, for others not (e.g. Shopping marketplace). This means that image thumbnail may never be shown if more relevant annotation has a higher priority
- Annotations show the item id, annotation type, payload (full or light), the score, the kind of item (doc, entity,…)
- Annotations can be computed at query time (query dependant) or be pre-computed and stored in an index
- If a highest ranked annotation does not meet a certain threshold, no annotations will be shown.
- Annotations may be shown in different shapes and sizes, with user interactions needed, or not.
How Are Annotations Chosen?
Annotations can come from one or more internal or external sources and the annotations can be scored using the framework described in the patent described in the current article.
A machine-learned scoring algorithm may rank all annotations for an item (e.g. web document).
Then, depending on the size of the screen of a given device (viewport), the system may display lightweight annotations or full annotations.
For example, a space-constrained mobile device may show the lightweight annotation and any devices larger than a laptop may be exposed to full annotations.
Annotation Pipeline
Here is the process by which Google may annotate search results:
- The query engine processes the query and determines what indexes to search from (web, images, hotels, marketplace, etc.).
- It ranks documents of a primary repository based on the query.
- It determines if it need to look at offline annotations, or need to process the annotation at query-time.
- Then, one or more annotators produce the annotations for each item using secondary repositories.
- The annotator(s) may rank annotations against each other using machine learning.
- The models can be trained using human raters where raters score the usefulness of the annotation.
- The annotator engine aggregates and ranks scored annotations produced by the annotator
- The query system then defines the number of items, the size and the way the annotations will be shown to the user.
What Does an Annotation Look Like?
Annotations are appending data to a search results.
Here an example of how an annotation may look like.
The table below shows the data that is stored with the annotation.
| Annotation | Description |
|---|---|
| Item_ID | Identifies the item in the first repository the annotation applies to |
| Type | Annotator type (Editorial, pros/cons, …) |
| Payload | Includes the data to be included in an annotation. (lightweight/full, instructions to display items). Dependent on the annotation type. |
| Score | Score of annotation. Could also be annotation features |
| Annotation identifier | What kind of item it identifies (document, product, entity,…) |
Search Engine Environments Where Auxiliary Information may be Taken From
The patent mentions a few kinds of search engines where the current patent may gather auxiliary information to fill-up annotations.
- Web search engine
- Media search engine (images)
- Marketplace search engine
- Travel Search Engine
- New Home Search Engine
- Entity Search Engine
Additional Annotation Information on Images
An additional patent named “Clustering Queries for Image Search” lists annotation information that is stored for images, it is unclear if and how this would be used in the context of the “Search Results Annotations” patent.
Image annotation information that is listed in the patent as being included is:
- titles,
- descriptions,
- ratings,
- Sources for the images,
- geographic locations at which the images were captured,
- Information provided by web page administrators, image providers or third parties.
- Advertising information
Definitions
| Patent term | Definition |
|---|---|
| Annotations | Auxiliary data about search results that are not part of the fundamental items (link, title and description) usually included in a search result |
| Items | Elements of different formats that can be contains in an index. (document, product, book, application, hotel, restaurant,…) |
| Query dependant annotation | Annotation that has to be processed at query time since it uses parts of the query to generate annotations |
| Offline annotation | Pre-computed annotations that are stored and assigned prior to a query |
| Annotator | Software that generates the annotation payload of an item for the search engine |
| Item repository | Items from a specific search engine corpus (web, images, marketplace) |
| Primary Repository | Repository defined as the primary corpus by the query engine based on the query |
| Secondary Repositories | Additional repositories used to provide annotations for the ranked items of the primary repository |
Google Search Infrastructure Involved
The “Search Results Annotations” patent mentions these elements from the Google Search Infrastructure:
- Query System
- Query Engine
- Annotator Engine
- Indexing Engine
Patent Details
| Name | Search Results Annotations |
| Assignee | Google LLC |
| Filed | 2021 |
| Status | Pending |
| Appl No. | 17/303,206 |
| Inventor(s) | Grzegorz Glowaty and al. |
| Patent | US20210406268A1 |
Other Similar Patents
- Automatic annotation for training and evaluation of semantic analysis engines (2013)
- Refining image annotations (2014)
- Methods, systems, and media for presenting related media content items (2020)
Conclusion
This patent was very interesting to understand what is going on when Google modifies the appearance of search results without the website owners specifically requesting those changes.
SEO Strategist at Tripadvisor, ex- Seek (Melbourne, Australia). Specialized in technical SEO. Writer in Python, Information Retrieval, SEO and machine learning. Guest author at SearchEngineJournal, SearchEngineLand and OnCrawl.