How to set a custom referer in wget

Some download endpoints check the HTTP Referer header before they return the requested file. Adding a custom referer in wget makes the request look like it came from the expected page without changing the rest of the transfer flow.

GNU wget sends that header with --referer. The same option works with spider checks and normal downloads, so the request can be inspected first and then repeated on the real transfer once the header value looks correct.

A referer check is often only one part of the remote policy. Some sites also require cookies, a custom User-Agent, or an authenticated session, so keep the header set minimal and confirm that the saved file is the expected payload instead of an access-denied page.

1. Probe the request in spider mode so the debug output shows the exact Referer: line before any file is written.

   $ wget --debug --spider --referer='https://www.gnu.org/' https://www.gnu.org/licenses/gpl-3.0.txt
   Setting --spider (spider) to 1
   Setting --referer (referer) to https://www.gnu.org/
   ##### snipped #####
   ---request begin---
   HEAD /licenses/gpl-3.0.txt HTTP/1.1
   Host: www.gnu.org
   Referer: https://www.gnu.org/
   User-Agent: Wget/1.25.0
   Accept: */*
   Accept-Encoding: identity
   Connection: Keep-Alive
   
   ---request end---
   HTTP request sent, awaiting response...
   ##### snipped #####
   200 OK
   Remote file exists.

   The request block confirms the exact referer value that wget is sending.

2. Repeat the same referer on the real download command once the request header looks correct.

   $ wget -O gpl-3.0.txt --referer='https://www.gnu.org/' https://www.gnu.org/licenses/gpl-3.0.txt
   ##### snipped #####
   HTTP request sent, awaiting response... 200 OK
   Length: 35149 (34K) [text/plain]
   Saving to: 'gpl-3.0.txt'
   
   ##### snipped #####
   
   'gpl-3.0.txt' saved [35149/35149]

   Quote the referer when it contains query strings, ampersands, or other shell-sensitive characters.

3. Check the saved file so a blocked HTML page is not mistaken for the intended payload.

   $ file gpl-3.0.txt
   gpl-3.0.txt: ASCII text

   A checksum or signature check is a stronger final verification when the site publishes one for the download.

4. Remove the sample file when the transfer was only a header test.

   $ rm -f gpl-3.0.txt