SEO automation with Python is a very popular topic right now. The easiest way you automate SEO with Python is by using crontab (cron) on Mac or Task Scheduler on Windows. In this guide, you will learn how to use crontab run your Python scripts automatically.
What Can You Automate?
You can extract data from Google Search Console and Google Analytics on a specific schedule, send yourself a weekly email report, log all your keywords every day or automate your keywords research, launch a crawl every week, save robots.txt file, and a lot more.
Before we can cover all that, we need to learn how to use Crontab.
What are Cron and Crontab?
Cron is the tool that let users run script, commands or software automatically on a specified schedule. Crontab is the file that lists the jobs that cron will be executing.
Cron doesn’t execute while the computer is asleep.
Cron is perfect to run simple task automation that can run during the day while you are working.
To automate task seamlessly whether your computer is open or not. Cron is not the right solution for you. You would need to run your scripts on servers like AWS for this kind of automation.
Overview of the Project
In this post, we will automate a very simple script to run every minute.
from datetime import datetime import os def write_file(filename,data): if os.path.isfile(filename): with open(filename, 'a') as f: f.write('\n' + data) else: with open(filename, 'w') as f: f.write(data) def print_time(): now = datetime.now() current_time = now.strftime("%H:%M:%S") data = "Current Time = " + current_time return data write_file('test.txt' , print_time())
This Python script will create a
test.txt and append the file with the time when the cron job executed.
What You Will Need?
This tutorial assumes that you already have installed:
- MacOS (if not, see the Windows Task Scheduler tutorial linked in the introduction)
- Code editor (like VScode)
And that you have at least basic knowledge of:
If not, start from the beginning of my Python for SEO tutorial.
How to Schedule a Task With Cron
Cron uses schedule expressions to know when you want to execute a task.
It has 5 parameters (
* * * * *) to let you choose when you want to run your script:
- the minute,
- the hour,
- the day of the month,
- the month,
- the day of the week.
CRON Schedule Expressions
The easiest way to create crontab schedule expressions is to use crontab.guru.
Also, here is a table with all the most common expressions that you will encounter.
|once on system startup|
|once a month|
|once a week|
|once a day|
|once a day at midnight|
|once an hour|
|once a minute|
|once every day of the week|
|once every specific day, at a specific time (Sunday at 12:30)|
How to Schedule a Script Using Cron?
To schedule a python script execution using cron, we will:
- Create the Python Script
- Create a crontab file
- Create a Python Script Sheduler
- Write the crontab
Create the Python Script
Using the script above, create a
.py Python script and save it to a location that you have permission (see Troubleshooting to understand why). In my case, I created an “Automation” folder under the
Create Your First Cron Job in Crontab
We will create and run a crontab file using the command line.
There are three (3) available commands to work with the crontab file:
crontab -l (to display),
crontab -e (to create) and
crontab -r (to delete).
List the current crontab
Check if you already have a crontab created.
-l means that you want to open the crontab file in the “list” mode.
If you have never created a crontab before, you will get this message:
crontab: no crontab for user.name.
Create a new Crontab
Now, we’ll add a new job to
-e means that you want to open the crontab file in the “edit” mode.
This will open the vi editor. Don’t be scared, I will walk you through it.
Vi Editor Commands
First, let’s look at the commands you can use in the vi editor.
|i||Switch to Insert mode (editing mode)|
|esc||Exit the editing mode|
|dd||Delete the current line|
|u||Undo the last change|
|:q!||Close the editor without saving changes.|
|:wq||Save the text and close the editor|
|→ + ||Move Cursor Faster|
|$||Move to end of line|
Create a Python Script Scheduler
We will now add the Python script scheduler command to in the crontab using the VI Editor.
How to Schedule the Python Script
The command we will use to execute the python script goes like this.
* * * * *is the crontab schedule expressions to schedule the script to run every minute;
cdis the command-line argument to tell crontab where the executable file is;
/Users/user.name/Automationis where my python script is located;
/usr/bin/pythonis where python is installed. You can find where you have installed it using the
whereis pythonor the
which pythoncommands in Terminal.
test_cron.pyis the name that I have given to the script share above.
Run the Script Automation
In the vi editor.
Switch to Insert mode by pressing
i to edit the file.
Add the command to execute the Python script.
* * * * * /usr/bin/python /path/to/file/<FILENAME>.py /path/to/file/<FILENAME>.log
esc to exit the editing mode.
:wq to save and quit the file.
w means to write and
q means quit.
After writing the crontab, you will get this message
crontab: installing new crontab which tells you created the crontab.
If all went well, you should now have the test.txt file in your folder with the time being added every minute (if not, see Trouble Shooting).
Wow, that was a lot.
Put simple, here is what you do:
- Create your Python Script;
- Open Terminal;
crontab -eto create crontab;
ito launch edit mode;
- Write the schedule command
* * * * * /usr/bin/python /path/to/file/<FILENAME>.py;
escto exit edit mode;
:wqto write your crontab
- To delete the running job:
- To delete the entire crontab: Run
- To delete a single cron job: Go to
crontab -e, press
:wqto write the file.
- To delete the entire crontab: Run
Whenever you get an error with a cron job, you will have a mail in terminal saying “
You have mail in /var/mail/user.name“. You can read the mail using the
Look at the message that you get to learn about the error.
To delete a message use
d, or to delete all use
delete *. To quit the mail, use
Operation not permitted
You might encounter the “[Errno 1] Operation not permitted” in MacOS when you try to run a cron job.
can't open file '<FILENAME>.py': [Errno 1] Operation not permitted
Here is how to Fix “Operation not permitted” Error in Terminal for Mac OS.
You will need to give full disk access to the terminal and to CRON like in this solution.
- In the Apple menu, choose
Privacy and Security
- Go to
Full Disk Accesson the left-hand side of the panel
- Click the lock to make changes
- Click on the Plus (
- Go to
- Click on the (+) sign again;
Command + Shift + Gand move to
- Restart the Computer
Cron needs access to the folder location that you work in. A lot of companies will restrict access to locations of your desktop or will have your
Documents folder in an internal server to keep a backup of your desktop.
IOError: [Errno 13] Permission denied: 'test.txt'
Here is how to fix the “Permission denied” error.
Just move your script to the user path
/User/user.name to see if you still get the error running the script from that location.
Kill a Currently Running Cron task
Let’s say you have created a Python script and ran it daily using CRON, but unfortunately, an error in your code created an infinite loop. How can you stop it without shutting down your computer?
You can start by finding any python script currently running. In Terminal type:
$ ps aux | grep python
Now, you will get a few lines that have this structure:
user.name 12345 0.0 0.0 1727387347 90 s004 S+ 2:25pm 0:00.00 grep python user.name 23456 0.0 0.0 111111111111111 294 s005 S+ 2:27pm 0:00.00 grep python
The first number you see on each line is your process ID. Above I have two PID:
If I want to kill the first process, I would use this line in Terminal:
$ kill -9 12345
That’s it! You have learned how to run a cron job on mac to automate your SEO with Python.
Sr SEO Specialist at Seek (Melbourne, Australia). Specialized in technical SEO. In a quest to programmatic SEO for large organizations through the use of Python, R and machine learning.