What is Rendering (how Googlebot Renders a Web Page) ?

Share this post

This post is for technical SEOs that want to understand how Googlebot renders a website.

I will help you understand what is rendering and show you in detail how chrome renders a web page as Googlebot uses evergreen Chromium renderer.

If you want to learn more, a lot of this content comes from Martin’s Splitt conference on Web Rendering Services at TechSEOBoost 2019.

You can also look at this presentation by Erik Hendriks, Software Engineer at Google to get some of the most common pitfalls of rendering.

What is Rendering?

According to Martin Splitt at Google, rendering is the process of turning (hyper)text into pixels.

L’attribut alt de cette image est vide, son nom de fichier est image-6.png.
Source: TechSEOboost 2019

How Does a Browser Display a Website to a User?

For a browser to display a web page to a user, it follows a few steps.

  1. Sends an HTTP Request to the server
  2. Parse the documents received by the server
  3. Create a Layout from the elements of the DOM tree.
  4. Paint the HTML

HTTP Request

The browser makes an HTTP Request to the server and the server sends data back to the browser.

Structure of an HTTP request in Chrome

In the case of a web page, the data sent back to the browser is HTML.

Then, the browser parses the HTML file.

and show the page to the user.

Parsing of the HTML

When a browser renders an HTML file to the user, takes each element one by one and parse it into DOM tree.

What is Parsing

Parsing means to analyze a sentence or a document into its parts. Parsing of an HTML means to analyze one by one each element of the HTML to understand their syntactic roles.

Simply put, parsing the HTML is to understand the structure of the HTML file.

How Chrome Parse the HTML File?

When Chrome Parse the HTML, it sends events to load each element into a DOM tree.

The Document Object Model, or DOM, represents the structure of the content.

Parsing of the HTML: Performance Report in Chrome

As you can see in the picture above, Chrome uses events to parse the HTML. It sends an event when the DOM starts to load, another one to load the DOM, and another one when it ends.

In the performance report, the Onload event is shown by the red line and the DOMContentLoaded Event is show by the blue line.

Note: Make sure that you load all critical content before the DCL event.

Creating the HTML Layout

After the Parsing of the HTML, Chrome knows the structure of your page (what is the H1, H2, where are your p elements, etc.)

Source: Martin Splitt

Now, Chrome will need to get the layout of the page. It maps each element to a specific position in a browser.

HTML Layout (w3Schools)

During the layout-ing, the browser is busy with the task and can’t do anything else. This is where the user has no choice but to wait.

These are the red bars that you can see in the graph.

If you continuously do layouting by moving things around in the page inefficiently, that’s a bad idea.

Martin Splitt (Google) – TechSEO Boost 2019

Painting of the HTML

Now, Chrome knows where each box goes and what content goes in each box.

Chrome will then paint the HTML tree with pixels. This is where Chrome adds the content to your page. It writes text, loads your images, adds colors, etc.

This is the step where (in Lighthouse) you start to see stuff like “First Paint”, “First Contentful Paint”, “First Meaningful Paint” and “Largest Contentful Paint” (shown by the acronyms in the image below).

Painting of a Web Page.

The thing is, painting requires Graphical Processing Units (GPUs), which is expensive. Also, Googlebot doesn’t really need to show pixels of the page to understand our content.

The Place of JavaScript in Rendering

JavaScript can change the content of the page. Without this possibility, single-page JavaScript websites wouldn’t exist.

Javascript can add, remove or change elements from the DOM tree.

If the changes are too big to fit in the initial layout, it can force Googlebot to re-layout, which as we have seen is bad for the user.

Re-layout after JavaScript DOM changes

You can look at some JavaScript functions for SEO to understand how you can change the DOM tree with JavaScript.

To find out how much content is loaded via JavaScript, you can compare the HTML code when you view-source (ctrl+U) versus what you can find in the DOM (ctrl+I)

Indexing Pipeline

Martin Splitt explains that the indexing process is made of many microservices that talk to each other and provide this screenshot.

  • The dup eliminator processes canonicalization of pages;
  • The robots parser analyses the robots.txt file;
  • The crawler crawls the web by following links;
  • The web rendering service
  • The link parser is used by the crawler to list and “click” on all links found on the page
  • The Content Parser serves to interpret the syntax of the content.

Web Rendering Service (WRS)

An amazing part of the Googler’s presentation is when he talked about the Web Rendering Service.

In this part, he talks about some cryptic developer stuff like Puppeteer, Service Wrapper and Shadow DOM… oooooouuuuuuhh!!!

I will not go into all the details of it.

Instead, just note these interesting points about WRS:

  • The indexing pipeline calls WRS, which renders;
  • WRS skips the pixels;
  • WRS makes connection requests to read HTML, CSS and JS files using single connections. Each connection is what we call the “crawl budget”;
  • WRS relies heavily on the cache and has certain rules to timeout if a page takes too much time to load.
  • WRS is used by all testing tools (mobile-friendly test, page speed, etc.). In exception, it will have an aggressive timeout and will not cache the page.
  • To render a page with Python or JavaScript, you can use the open-source Headless chromium renderer, Rendertron.
  • WRS is stateless (cookies are overridden after each page load).

Other interesting features of Web Rendering Service are explained by Jamie Alberico.

  • WRS is the name that represents the collective elements used in Google’s rendering service.
  • WRS reads HTTP response and header;
  • Googlebot’s first priority is not to degrade the user experience so it uses a “crawl rate limit”.

The end result would look something like this.

How Can You Improve Rendering for SEO?

A lot of stuff can help your crawl budget and the rendering of the page. Remember this, what is good for the user, Googlebot will like it.

Here are a few advises that Martin Splitt gave:

Tip #1: Make sure that DOM changes made with JavaScript don’t force Chrome to recreate the HTML layout.

Tip #2: Use Tree shaking to remove unused code

Tip #3: Use simple and clean caching by adding content caches hashes. You can do this by using Webpack.

Tip #4: Bundle your code, but split it into reasonable bundles.

This is it!

Other Useful SEO Content on Rendering

A very special thanks to Martin Splitt for sharing this amount of useful information on what is rendering and how Google renders a website.