How to set a custom referer in Wget

Some servers check the Referer header to control access. By specifying a custom referer, you can simulate coming from a particular page, enabling downloads that would otherwise be blocked or altered based on the perceived origin.

wget allows adjusting the referer easily. This feature helps bypass basic restrictions or replicate browser navigation patterns, enabling automation of protected downloads. However, always respect the website’s policies and ethical guidelines.

Custom referers are useful for scraping resources that only load when the request appears to originate from a specific page, ensuring reproducible and predictable data retrieval.

1. Open the terminal on your system.
2. Use the --referer option followed by the URL you want to send as the referer.

   $ wget --referer=http://example.com/referer-page/ http://targetwebsite.com/download/file.zip
   --2024-09-16 13:45:05--  http://targetwebsite.com/download/file.zip
   Resolving targetwebsite.com (targetwebsite.com)... 93.184.216.34
   Connecting to targetwebsite.com (targetwebsite.com)|93.184.216.34|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 19834 (19K) [application/zip]
   Saving to: ‘file.zip’
   
   file.zip          100%[===================>]  19.36K  --.-KB/s    in 0s    
   
   2024-09-16 13:45:06 (421 MB/s) - ‘file.zip’ saved [19834/19834]

   This command sets the referer to http://example.com/referer-page// and downloads the file from http://targetwebsite.com//.

3. Verify the downloaded content for accuracy and completeness.
4. Combine the --referer option with other Wget options if needed, such as setting a user agent or additional headers.

   $ wget --referer=http://example.com/referer-page/ --user-agent="Custom User Agent" --header="Custom-Header: Value" http://targetwebsite.com/download/file.zip

   This command combines the custom referer with a custom user agent and additional header.

5. Use this command within scripts or automation tasks for repetitive actions.
6. Ensure compliance with website policies when using these options.