How to use download compression in wget

Compressing responses reduces bandwidth usage and speeds up downloads from servers that support compressed content. wget can request and handle compressed data seamlessly, improving efficiency when working with large files, JSON APIs, or slow connections.

By default, wget supports gzip and deflate if the server provides them. Sending an Accept-Encoding header requests compressed content. Once received, wget automatically decompresses the response, delivering the original uncompressed file.

Controlling compression allows you to balance efficiency and compatibility. While automatic decompression simplifies usage, disabling it preserves the compressed form for manual handling when needed.

1. Use the --header option to request compressed data from the server.

   $ wget --header="Accept-Encoding: gzip, deflate" https://www.example.com/data -O data.json
   --2024-12-10 10:04:00--  https://www.example.com/data
   HTTP request sent, awaiting response... 200 OK
   Length: unspecified [application/json]
   Saving to: ‘data.json’

   wget will decompress the file automatically if the server sends compressed content.

2. Verify that the resulting file is in readable format after automatic decompression.

   $ file data.json
   data.json: ASCII text

3. Save compressed responses as-is by naming the output with a .gz extension and not requesting automatic decompression.

   $ wget --header="Accept-Encoding: gzip" https://www.example.com/data -O data.gz

   You can manually decompress this file later using gunzip.

4. Enable automatic compression handling without manually setting headers by using %--compression=auto.

   $ wget --compression=auto https://www.example.com/data -O data.json

   The %–compression=auto option ensures wget requests and handles compressed content automatically.