How to download a whole website using Wget

Downloading entire websites can be essential for various reasons, such as backing up content, offline browsing, or mirroring sites for hosting elsewhere. wget, a powerful command-line tool available in many UNIX-like operating systems, offers a convenient way to download websites in their entirety.

With its wide range of options, wget ensures that you can customize your downloading experience. For example, you can choose to retrieve only specific file types, follow or ignore specific links, or control the depth of your crawl. By default, wget fetches a page and all its components, ensuring that the downloaded content looks the same as online.

However, it's crucial to use wget responsibly. Mass downloading can put unnecessary stress on servers and potentially violate website terms of service. Before using wget to download an entire site, ensure you have permission to do so.

1. Open the terminal.
2. Install wget if not already available.

   $ sudo apt update && sudo apt install wget #Ubuntu and other Debian derivatives

3. Begin downloading the desired website.

   $ wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://www.example.com/

   Option	Description
   --mirror	Tells wget to mirror the site
   --convert-links	Adjusts links for offline viewing
   --adjust-extension	Adjusts the file extensions
   --page-requisites	Downloads all page prerequisites
   --no-parent	Avoids downloading links outside of the specified domain
   --no-check-certificate	Skips certificate checks (use with caution)
   --limit-rate=200k	Limits the download speed to 200KB/s

4. Monitor the download progress in the terminal.
5. Once the download completes, navigate to the directory where you initiated the command to access the downloaded content.
6. Explore the downloaded site offline using any web browser by opening the main HTML file.

To respect the robots.txt rules, use the following command with –execute robots=off removed:

$ wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains yourdomain.com --no-parent www.yourdomain.com --execute robots=off

Enjoy your offline content and always remember to use tools like wget responsibly!