How to set a custom referer in wget

Some download endpoints check the HTTP Referer header before they return a file, especially when the URL is expected to be reached from a product page, a portal view, or a download landing page. In those cases the URL alone is not enough, and a direct wget request can fail even when the file path is otherwise correct.

Wget handles that requirement with --referer, which adds a Referer: header to the request while leaving the rest of the transfer logic alone. That keeps redirects, HTTPS negotiation, resume support, and output handling the same as a normal download while still matching the server-side check that looks for a specific source page.

Referer checks are often only one piece of a broader gate. Some endpoints also inspect cookies, the User-Agent header, or language hints, so a correct referer does not guarantee a 200 OK on its own. Use the smallest header set that satisfies the remote policy and avoid pretending to be an unrelated client unless the workflow genuinely requires it.

1. Confirm the exact Referer header value with a low-cost echo request before trying the real download URL.

   $ wget -qO- \
     --referer='https://example.com/download-page/' \
     https://httpbin.org/headers | jq -r '.headers.Referer'
   https://example.com/download-page/

   Use the exact scheme, host, path, and trailing slash pattern that the remote endpoint expects, because small referer mismatches can still trigger a block.

2. Probe the target URL with --spider and the same referer before pulling the full file.

   $ wget --spider --server-response \
     --referer='https://example.com/download-page/' \
     https://httpbin.org/robots.txt 2>&1 | sed -n '1,14p'
   Spider mode enabled. Check if remote file exists.
   --2026-03-27 07:04:14--  https://httpbin.org/robots.txt
   Resolving httpbin.org (httpbin.org)... 32.194.43.65, 52.71.170.232, 54.172.102.128, ...
   Connecting to httpbin.org (httpbin.org)|32.194.43.65|:443... connected.
   HTTP request sent, awaiting response...
     HTTP/1.1 200 OK
     Date: Thu, 26 Mar 2026 23:04:15 GMT
     Content-Type: text/plain
     Content-Length: 30
   Length: 30 [text/plain]
   Remote file exists.

   Spider mode is a cheap validation pass for referer-gated URLs because it confirms acceptance without committing to the full download first.

3. Add more headers only when the endpoint checks more than referer alone.

   $ wget -qO- \
     --referer='https://example.com/download-page/' \
     --user-agent='Mozilla/5.0 (X11; Linux x86_64)' \
     --header='Accept-Language: en-US,en;q=0.9' \
     https://httpbin.org/headers | jq '{Referer: .headers.Referer, "User-Agent": .headers["User-Agent"], "Accept-Language": .headers["Accept-Language"]}'
   {
     "Referer": "https://example.com/download-page/",
     "User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
     "Accept-Language": "en-US,en;q=0.9"
   }

   Quote the referer when it includes query strings or other shell-significant characters.

   Related: How to change the user agent in wget
   Related: How to use custom cookies with wget

4. Download the file with the validated referer and confirm the result looks like the expected payload.

   $ wget -O robots.txt \
     --referer='https://example.com/download-page/' \
     https://httpbin.org/robots.txt 2>&1 | sed -n '1,10p'
   --2026-03-27 07:04:14--  https://httpbin.org/robots.txt
   Resolving httpbin.org (httpbin.org)... 32.194.43.65, 52.71.170.232, 54.172.102.128, ...
   Connecting to httpbin.org (httpbin.org)|32.194.43.65|:443... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 30 [text/plain]
   Saving to: 'robots.txt'
   $ file robots.txt
   robots.txt: ASCII text

   Final verification should include both a normal HTTP response and a quick file-type or content check so a saved login page is not mistaken for the real download.