Get Started With Selenium and Python Automation

Share this post

This post is part of the complete Guide on Python for SEO

I made this easy tutorial to help non-developers install Selenium Webdriver using Python. Selenium Webdriver is an amazing tool for SEO automation.

If you haven’t installed Python yet, make sure that you install Python using Anaconda.

Install Selenium

Let’s get started by installing Selenium.

Install Selenium With Anaconda

If you have installed Python with Anaconda, go to Anaconda Prompt from your navigation.

Type in the following command.

conda install -c conda-forge selenium

Now, a line will come up asking you to proceed. Press “Y” and click enter.

Install Selenium With Pip

If you need to install Selenium with Pip, use this in your command line.

pip install Selenium

Now that we have Selenium installed, we need to install a webdriver.

Install Selenium Webdriver on Chrome

Selenium needs a webdriver to work with your browser.

Step 1: To install Selenium Webdriver, just go to Seleniumhq.org and go to “Third Party Browser Drivers” section.

Step 2: Click on “Google Chrome Driver” . If you use Mozilla, just install Mozilla GeckoDriver (or any other browser that you use).

Step 3: Click on the latest stable release.

Download Latest stable release of Selenium

Step 4: Download the correct version for your operating system. I am on Windows, so I’ll select chromedriver_win32.zip.

Step 5 Unzip the folder

Step 6: Go back to Anaconda Prompt and type python.

Step 7: Import Selenium Webdriver

In your command-line, or IDE, write this command.

from selenium import webdriver

If everything was properly installed, you should get something like this.

You Might Also Like Get BERT Score for SEO (by Pierre Rouarch)

If you get an error message, it might be that you have forgot to type “Y” when asked to “proceed”.

Go ahread and try to reinstall Selenium in conda and read the instructions properly.

If still not working, head over to Stackoverflow to find a solution.

Next, we’ll set-up our browser.

Make Your First Automated Task With Selenium

To make your first automated task with Selenium, you will need to find the executable path of your chrome browser.

To find it just chrome://version in your browser.

Open And Close Your Browser

To open and close your browser using selenium, you will be using the webdriver.Chrome() and the quit() functions.

import time
from selenium import webdriver

driver=webdriver.Chrome(executable_path=r"C:\Users\j-c.chouinard\Downloads\chromedriver_win32\chromedriver.exe") #Path to Chrome Driver
time.sleep(5) # Wait for 5 seconds, so that you can see something.
driver.quit()

A browser will open with some kind of mention that Chrome is being controlled by automated test software and then it will close.

Note that the path is optional, but it is preferable to add it to make sure that the script works.

Open a Website With Selenium

Opening a website in your browser is simple using the get() function.

from selenium import webdriver

driver=webdriver.Chrome(executable_path=r"C:\Users\j-c.chouinard\Downloads\chromedriver_win32\chromedriver.exe") #Path to Chrome Driver
driver.get('https://www.jcchouinard.com/')

How to Open a Link

There are many ways that you can open links automatically with Selenium. The process is this one:

Find an element and store it in a variable
Get the href attribute
Click the link

There are multiple ways that you can find an element.

#1. Find an element by link text and store it in a variable 
driver.get('https://moz.com/')
linkText=driver.find_element_by_link_text("Blog")
#Access the "text" attribute.
linkText.text
##Out[23]: 'Blog'

Let’s get the href attribute using get_attribute.

driver.get('https://moz.com/')
linkText=driver.find_element_by_link_text("Blog")
linkText.get_attribute("href")
##Out[24]: 'https://moz.com/blog'

Click the link using click().

#Open the webpage
driver.get('https://moz.com/')

#Add link element to a variable
linkText=driver.find_element_by_link_text("Blog")

#Wait 1 Second and click the link
time.sleep(1)
linkText.click()

#Wait 1 second and close the browser
time.sleep(1) 
driver.quit()

Congratulations! You are now ready to learn SEO reading Moz’s blog.

You Might Also Like BrightonSEO 2019 Slides & Videos

How To Use a Search Box

To perform a search using a search box using Selenium, you will need to get the Id of the input and of the click button.

To do this, right-click on the search box and click inspect element.

Note the ID of the Input element.

Repeat the process to find the ID of the button.

#Get to your page
driver.get('https://www.jobillico.com/')

#Add the ID of the search Box
searchBox=driver.find_element_by_id("search-engine-keyword-job")

#Add the search text
searchBox.send_keys("seo")

#Get Button ID
submit=driver.find_element_by_id("btn-search-engine")

#Click The Button
submit.click()

You could also use Keys to submit the form. This will simulate a user that presses the ENTER key.

from selenium.webdriver.common.keys import Keys
driver.get('https://www.jobillico.com/')
searchBox=driver.find_element_by_id("search-engine-keyword-job")
searchBox.send_keys("seo")
searchBox.send_keys(Keys.ENTER)

These are some of the many ways that could work with any form. The next example uses another way using the submit()function instead.

Search For Something On Google Using Selenium

You can easily search for anything on Google using Python and Selenium. By typing this bit of code, it will:

Open a Web Browser
Go to Google.com
Search for a Query String
Close the Browser

All of this will be done at a certain rate because of the time.sleep function.

import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

driver=webdriver.Chrome(executable_path=r"C:\Users\j-c.chouinard\Downloads\chromedriver_win32\chromedriver.exe") #Path to Chrome Driver
driver.get('http://www.google.com/');
time.sleep(5) 

#Search and submit
search_box = driver.find_element_by_name('q')
search_box.send_keys('jcchouinard')
search_box.submit()

#Wait for the user to see something and then close
time.sleep(5) 
driver.quit()

Open The First Result In Google With Python

The first result in Google is of great interest to SEOs, so you might want to go and open the first link in Google.

To do this, the link name is not a good selector since it always changes.

We will need some Xpath to perform this task.

You Might Also Like SEO Split-Testing Experiments Using Google Tag Manager

Even if you know nothing about Xpath, you can reuse some xpath functions for Google that I have created for you.

import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

#Open Web Driver
driver=webdriver.Chrome(executable_path=r"C:\Users\j-c.chouinard\Downloads\chromedriver_win32\chromedriver.exe")

#Go To Google .com
driver.get('https://www.google.com/');
time.sleep(1)

#Search For a Query
search_box = driver.find_element_by_name('q')
search_box.send_keys('jcchouinard')
search_box.submit()

#Select First Result Using Xpath
firstResult=driver.find_element_by_xpath('(//div[contains(@class,"r")]/a[contains(@ping,"/url")])[1]')

#Click On The Link
firstResult.click()

Enjoyed this post? Tell me how you automate your SEO using Selenium webdriver and Python.

Jean-Christophe Chouinard

Sr SEO Specialist at Seek (Melbourne, Australia). Specialized in technical SEO. In a quest to programmatic SEO for large organizations through the use of Python, R and machine learning.