How to set a custom referer in wget

Some download endpoints check the HTTP Referer header before they return the requested file. Adding a custom referer in wget makes the request look like it came from the expected page without changing the rest of the transfer flow.

GNU wget sends that header with --referer. The same option works with spider checks and normal downloads, so the request can be inspected first and then repeated on the real transfer once the header value looks correct.

A referer check is often only one part of the remote policy. Some sites also require cookies, a custom User-Agent, or an authenticated session, so keep the header set minimal and confirm that the saved file is the expected payload instead of an access-denied page.

Related: How to send custom headers with wget
Related: How to change the user agent in wget
Related: How to use custom cookies with wget

Steps to set a custom referer in wget:

  1. Probe the request in spider mode so the debug output shows the exact Referer: line before any file is written. 
    $ wget --debug --spider --referer='https://www.gnu.org/' https://www.gnu.org/licenses/gpl-3.0.txt
Setting --spider (spider) to 1
Setting --referer (referer) to https://www.gnu.org/
##### snipped #####
---request begin---
HEAD /licenses/gpl-3.0.txt HTTP/1.1
Host: www.gnu.org
Referer: https://www.gnu.org/
User-Agent: Wget/1.25.0
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
##### snipped #####
200 OK
Remote file exists.

    The request block confirms the exact referer value that wget is sending.

  2. Repeat the same referer on the real download command once the request header looks correct. 
    $ wget -O gpl-3.0.txt --referer='https://www.gnu.org/' https://www.gnu.org/licenses/gpl-3.0.txt
##### snipped #####
HTTP request sent, awaiting response... 200 OK
Length: 35149 (34K) [text/plain]
Saving to: 'gpl-3.0.txt'

##### snipped #####

'gpl-3.0.txt' saved [35149/35149]

    Quote the referer when it contains query strings, ampersands, or other shell-sensitive characters.

  3. Check the saved file so a blocked HTML page is not mistaken for the intended payload. 
    $ file gpl-3.0.txt
gpl-3.0.txt: ASCII text

    A checksum or signature check is a stronger final verification when the site publishes one for the download.

  4. Remove the sample file when the transfer was only a header test. 
    $ rm -f gpl-3.0.txt