How to download from a list in Wget

When downloading multiple files from the internet, rather than fetching them one by one, a more efficient approach is to use a list. Wget, a popular command-line utility, provides this functionality through its native support for reading URLs from a file.

This method ensures that you can manage and maintain a collection of links in a single file, making it easier to initiate multiple downloads with a single command. Whether you're maintaining backups, setting up a local repository, or simply archiving data, downloading from a list with Wget is an invaluable skill.

Using the Wget -i switch, you can specify a file that contains a list of URLs to be downloaded. This file should contain one URL per line.

1. Create a text file containing the list of URLs you want to download. Each URL should be on a new line.
2. Save the file with a memorable name, for instance, download-list.txt.
3. Open the terminal.
4. Navigate to the directory where download-list.txt is saved using the cd command.

   $ cd /path/to/directory

5. Run Wget with the -i option followed by the filename to start downloading.

   $ wget -i download-list.txt

6. Monitor the terminal for the download progress. Once complete, the files will be saved in the current directory.

   If any download fails, Wget will attempt to retrieve it until the list is fully processed.

7. For large lists or unstable connections, you can add the –continue switch to ensure Wget resumes incomplete downloads.

   $ wget --continue -i download-list.txt

   Related: resume-incomplete-downloads

8. To limit the download speed, use the –limit-rate option. For example, to limit the rate to 200k, use:

   $ wget --limit-rate=200k -i download-list.txt

   Be mindful of network usage, especially on shared or limited connections.

9. Once all downloads are complete, review the terminal output to ensure there were no errors.

$ wget -i download-list.txt -P /path/to/save/