In this tutorial, I will show you how to scrape Google Search Results using Apify.
Here is an example where I scraped 5 keywords for 0.02$.
Get FREE Apify Credits
Why Use Apify For Scraping Google?
Apify allows you to scrape Google without:
- the fear of getting your IP blocked AND
- the need to how to code AND
- the need to handle proxies
AND it is super easy to learn AND relatively cheap.
Is Apify Free?
Apify is not free, but does provide free credits for you to get started.
Overview of Apify Google Web Scraper
Apify has a built-in functionality for scraping Google. It is very easy to use and scrapes things like:
- Top Search Results Paid and Organic
- People Also Ask
- Related Queries
- Emphasized keywords
- Sitelinks
Customize Google Scraping Settings
With Apify‘s Google web scraper, you can also customized web scraping settings such as the country, the language or whether you want to scrape Desktop or Mobile search results.
Apify VS SEM Rush VS Ahrefs
While Apify may not be as complete as SEM Rush or Ahrefs, it has the upside of being way cheaper for people looking for small scale web scraping.
Apify also provide many web scraping tools that its competitor doesn’t such as proxies, scrapers for sites other than Google (e.g. Amazon, Twitter, TikTok, etc.)
How to Scrape Google with Apify
It takes around 5 minutes to create your Apify account, log-in, learn and to run your first Google web scraper.
Time needed: 5 minutes
- Sign-Up to Apify
- Log-in to Apify
Automatically, you get 5$ free credits and are ready to scrape Google.
- Open the Apify Store
On the left hand panel, click on the “store” button
- Find the “Google Search Results Scraper”
- Add the Keywords that You want to Scrape Search Results
Make a list of keywords that you want to scrape. 6 keywords will cost you around 0.02$
- Set-up the Number of Results to Scrape
The “max page per search term or URL” and “Results per Google Page” allows you to define the parameters to add in the Google URL that will tell how many search results to display. If you set this to 1 and 100 respectively, you will get the top 100 search results in a single HTTP request.
- Decide to Scrape Mobile or Desktop
You can toggle the “Mobile results” button to decide whether you want to scrape Google mobile or desktop. I usually do both.
- Decide the Location and Language From Where to Scrape
Google search results are not the same in the USA as they are in Italy. You can define the Geo and the language in the “Additional filters” section.
- Store an Example of the Search Results
If you want to come back and debug your SEO in the future, you probably will want to store a version of the HTML so that you can look at it in the future.
You can do so by going in “Advanced options (for developers)” and toggle “Save HTML to Dataset” and “Save HTML to key-value store”.
- Enhance Web Scraping Settings (optional)
- Run your Google Web Scraper
Once ready to scrape, you can click on “Save & Start”
Apify Scraped Search Results Dashboard
After Scraping Google Search results, you will have a dashboard showing you the scraped data from the SERP.
In the Overview tab, you get a table with the Search results where each search term is a row and each column tells you:
- Total number of results
- Related queries
- Paid results
- Paid products
- Organic Results
- People Also Ask.
You can expand the amount of available data by clicking on “all fields”.
This snapshot will give your the HTML and the snapshot URL, as well as few other fields.
You can choose to have the data as a table or as a JSON to your linking.
By clicking, each value of the table, you can see the underlying JSON scraped data.
If you go in the storage tab, you’ll find you SERP snapshots
As well as options to download and export the scraped Google search results data.
How to Save a Version of Google SERP with Apify
You can scrape Google and save a version of the HTML so that you can view the page in your browser in the future by using Apify Google Scraper and select “Save HTML to Dataset” and “Save HTML to key-value store” in the “Advanced options (for developers)” section.
How to Schedule Scraping Google Search Results with Apify
After you have run a Google scraper with Apify, you can schedule its execution so that you can have historical comparison for the search results.
To schedule a Google scraper with Apify, go in Actors > Runs and click on “Schedule your actor”.
In the pop-up you can set-up the scraper to run up to the minute (depending on your budget).
It is a good idea to create different scraper for different categories of keywords or site sections. This way you can diagnose whether certain site sections go better of worse over time.
Be mindful of your budget when setting this up.
What Other Google Products Can you Scrape with Apify
You can scrape many Google products using Apify such as:
- Google Maps
- Google Trends
- Google Shopping
- Google Play Reviews
Conclusion
This is the end of the introduction on scraping Google using Apify. You can also read how to scrape Google with Screaming Frog.
SEO Strategist at Tripadvisor, ex- Seek (Melbourne, Australia). Specialized in technical SEO. Writer in Python, Information Retrieval, SEO and machine learning. Guest author at SearchEngineJournal, SearchEngineLand and OnCrawl.
