How Google Displays Images Within Web Search Results?

Have you ever wondered how Google decides whether or not they should, or even where to show images in web search results?

In this post, I will cover how Google ranks what is often called the “Image Pack” in its regular web search results according to the specifications of the Google patent titled “Image display within web search results” by Grant Dasher and David R.H. Miller.

How Google Shows the “Image Pack” In Web Search?

In order to show the image search results within the web search results, Google needs to identify if:

1. the query is an image seeking intent,
2. there are enough high-quality images to satisfy the user,
3. users often search for the query in the Google image search engine

After, it will define the size of the image display module and then rank the module against regular search results based on:

• Query Intent Score
• Normalization Score
• Image Quality Score

When and Where Images are Shown in SERP

The image display environment and position relative to general search results varies in proportion to the inferred user interest in images and the quality of the images that are responsive to the query.

Source: Dynamic image display area and image display within web search results

Thus, if many high quality images are available, but there is little inferred interest in the images, then no image search results will be shown. Conversely, if there are few high quality images available, but there is a significant inferred interest in the images, then Some image search results will be shown.

How the Image Seeking Intent Score is Determined?

The image intent score is determined by the “Image Display Trigger Module” based on the search property ratio and the image CTR for a query.

The search property ratio is the proportion of the number of times the search query is submitted for a search in the image corpus versus the number of time it is submitted in the resource corpus.

Here are the factors that influence the image seeking intent score:

• Overall quality of images available for a query (topical relatedness and relevance score)
• Ratio of images to webpages available for a query
• Click through rates of images available for the query
• Ratio of number of searches for a query in the image corpus (Google Image) compared to the number of searches of the same query in the resource corpus (regular web search).
• Explicit image intent terms: Pictures, Photos, Diagrams, Images
• Explicit non-image intent terms: Articles, Dissertations, Studies

Typical case where image display environment would not show

A user searches for “SEO” on Google and there are few images labelled “SEO” compared to a lot of web pages talking about “SEO”. The quality of the images are not so great. Few people search in Google Image for “SEO”, and when they do, they don’t click much on the images. Image does not meet intent threshold.

Typical case where image display is inserted first and take a lot of space

A user searches for “travel images”. Every travel webpage have at least a few high quality images. People search as much for “travel images” in Google Search than in Google Images and the CTR in Google Images is very high. Image meets intent threshold.

Note that the query “travel” by itself still meets the intent threshold, but the image intent is less than for “travel images”.

The opposite is also True where the intent means that a query is not meant to show images.

What Determines the Size of the Image Display in SERP?

The size of the image display in the SERP is determined by the “Image Display and Positioning System” by calculating the image intent score and the flatness score.

The search system determines the dimension of the image display environment in the SERP.

The size of the display environment (or Real Estate available for images in the SERP) varies from query to query. It depends of the intent score of the query and the overall quality score of the images available.

• Higher intent score, higher quality images, the larger the dimension of the image display environment

The above explains why eCommerce and Travel queries tend to yield more image snippets in SERP, including multiple rows of image search results.

However, if intent score is high, but image quality is low overall, the number of rows may be lower.

What Determines the Position of the Images on SERP?

The position of the image pack in the SERP is determined by the insertion score.

Insertion scores depends on:

• Query Intent Score: Intent based on image quality and search and click data
• Query Normalization Score: Normalization of image score ensuring to process the placement of the image display environment in the same manner that it processes the placement of general search results. Normalized based on: uniformity of images using co-click data, avg. quality of images, length of query.
• Central tendency score of quality scores: Mean of the relevance score of the images

The greater the insertion score, the lower to position of the image pack. A very high score may bring the image pack to position 1 in the SERP.

The insertion score has to be greater than the relevance scores of the general search results.

Full Process

1. Receive Search Query
2. Check if confidence score threshold is met. Is it a long tail query? Is there sufficient historical data from which and image intent can be determined?
   1. If it is not met, search a subset of the image index (1-25% of the index) and see if query meets the second resource score threshold, or search a subset of combined image/search index to score the images. See [0057 to 0064]
   2. If it is met, determination of the image intent behind the query using the insertion system. It is the trigger module that checks the intent score of the query and compare it to intent threshold
3. In both cases above:
   1. if threshold is not met, it means that the query is not an image seeking intent, so don’t search the image index and generate search results
   2. If it is met, follow to step 4.
4. Search query in the resource corpus index using typical information retrieval score (relevance of a resource to a query relative to other resources) and optionally looking at the authority score of each resource.
5. For an image seeking query (query that meets the image intent threshold), the search systems accesses the image corpus index and requests the positioning module. Relevance to label associated with image and feedback score derived from CTR when image is shown in image search results
6. The image display and positioning module inside the insertion systems evaluate the size and position (insertion score) of the image display environment
7. The most relevant images are displayed in the image display environment

Second resource score example: If 1000 images of a subset come from 500 web pages. Out of those 500 pages, 300 include at least 1 image that exceeds the search score threshold. The second resource score is then 300/500 = 0.6

Definitions

Patent Term	What it is
Query Confidence Score	Measure of accuracy of the query search property ratio
Query Intent Score	Intent based on image quality and search and click data
Search Property Ratio	Image Search vs Web Search
Click Through Rate	Click/impressions on image corpus or any corpus
Insertion Score	Position to insert images in search result
Flatness Score	Distribution of image quality scores
Central tendency score	Mean of the relevance score of the images
Query Normalization Score	Puts image and web search queries on the same scale
Second Resource Score	Measure of relatedness of query to a subset of the image corpus. Similar to query Intent score.
First Resources	Search for web pages
Second Resources	Search for Images

Google Search Infrastructure Involved

The “Image display within web search results” patent mentions these elements from the Google Search Infrastructure:

• Search System
  • Search Engine
  • Image Search Results Insertion System
    • Image Display Trigger Module
    • Image Result Positioning Module
  • Search Results Processing Module
• Resource Corpus Index
• Image Corpus Index
• Historical Search Data
• Corpora Models

Patent Details

Below we are discussing the process of how images are shown in Google Search based on the patent:

Dynamic image display area and image display within web search results

Which was later updated to:

Image display within web search results

The update is mostly related to reducing the number of image search required in the process.

Name	Image display within web search results
Assignee	Google LLC
Filed	2013-03-12
Granted	2015-11-10
Status	Active
Expiration	2033-08-17
Application	13/795,397
Inventor	Grant Dasher, David R.H. Miller
Patent	US20150161268A1

Conclusion

The “Image display within web search results” patent provide good insights into how Google knows when to display images into regular web search, how prominent the image should be and what images should be shown.