How to download directories recursively using wget

Sometimes you need more than just a single file. Recursive downloads let you retrieve entire directories or site sections, preserving internal links, structure, and associated files.

wget supports recursion using --recursive and related options that control depth, file types, and excluded directories. With careful tuning, you can clone complex file hierarchies or partial websites efficiently and safely.

Be mindful of the website’s load and permissions. Overly broad recursion can stress servers and retrieve unneeded data. Adjust recursion depth, file filters, and wait intervals to behave responsibly.

1. Use --recursive to enable recursion.

   $ wget --recursive http://www.example.com/directory/

2. Limit recursion depth with --level to avoid excessive data.

   $ wget --recursive --level=1 http://www.example.com/directory/

   --level=1 restricts downloads to the starting directory and its immediate links.

3. Exclude certain directories with --exclude-directories.

   $ wget --recursive --exclude-directories=/private,/temp http://www.example.com/directory/

4. Add a delay between requests using --wait to reduce server load.

   $ wget --recursive --wait=2 http://www.example.com/directory/

   Introducing a wait prevents overloading the server with rapid-fire requests.

5. Filter file types with --accept for more targeted downloads.

   $ wget --recursive --accept=jpg,png http://www.example.com/directory/