In this post, we are going to see how to make a mirror of a web page or even your entire website with Wget and Python. Then, we will run the mirror on Ngrok to make a public URL.
That way you will be able to make changes and test it in any tool: Screaming Frog, GTMetrix, Google tools (lighthouse, mobile-friendly test, Rich Results,…), or most other tools.
![](https://www.jcchouinard.com/wp-content/uploads/2021/09/image-88-1024x354.png)
For example, I could test the structured data implementation before sending it to the web developers for implementation.
![](https://www.jcchouinard.com/wp-content/uploads/2021/09/image-89-1024x635.png)
You will need to install Python first.
Install Ngrok and Wget
On Mac
We will install both packages using Homebrew.
Open Terminal and run:
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install wget
$ brew cask install ngrok
On Windows
To install Ngrok on Windows according to Twilio.
- Download the ngrok ZIP file
- Unzip the ngrok.exe file
- Place the ngrok.exe in a folder of your choosing
- Make sure the folder is in your PATH environment variable
Read this guide to install Wget on Windows.
Extract Web pages with Wget
WGET is a free tool to crawl websites and download files via the command line. We are going to use it to download files from a website to make a copy on NGrok.
Extract a single Web Page
Open the Terminal and run:
$ wget -E -H -k -K -p --convert-links https://www.example.com/path
Extract Multiple URLs
Add all urls in a urls.txt
file.
https://example.com/1
https://example.com/2
https://example.com/3
Then, open the Terminal and run this command.
$ wget -E -H -k -K -p --convert-links -i urls.txt
Run a Python Server
$ python -m http.server
Run Ngrok
To run Ngrok on the server you created, you need to open another Terminal.
In the second Terminal, run this command.
$ ngrok http http://0.0.0.0:8000/
![](https://www.jcchouinard.com/wp-content/uploads/2020/12/image-2-1024x343.png)
Copy the forwarding URL in your browser and that’s it.
Conclusion
This is it. If you go to the Ngrok url, you now have a public URL that you can use to test in any tool.
![jean-christophe chouinard seo expert in quebec city](https://www.jcchouinard.com/wp-content/uploads/2019/11/jean-christophe-chouinard-seo-expert-quebec.jpg)
SEO Strategist at Tripadvisor, ex- Seek (Melbourne, Australia). Specialized in technical SEO. Writer in Python, Information Retrieval, SEO and machine learning. Guest author at SearchEngineJournal, SearchEngineLand and OnCrawl.