How to change user agent in cURL

The User-Agent is a header that tells a server information about the client making the request. It typically includes details about the operating system and browser. By default, cURL uses its own name and version as the User-Agent string, which can be identified by servers. Changing the User-Agent allows the client to mimic requests from different browsers or devices.

Servers often use the User-Agent string to deliver specific content. For example, mobile devices may receive optimized pages, while desktop browsers get more complex ones. Some servers block certain User-Agent strings, like bots or automated requests from cURL. Adjusting the User-Agent string can bypass these restrictions and access content otherwise unavailable.

Modifying the User-Agent string in cURL is useful for testing how websites behave for different clients. This allows you to simulate requests from specific browsers, devices, or bots, helping you troubleshoot issues or analyze server responses. It is a common practice for developers and testers to adjust the User-Agent for better control over HTTP requests.

1. Open the terminal.
2. Make an HTTP request using cURL to verify the default User-Agent.

   $ curl  http://www.example.com -v 2>&1 | grep '^>'
   > GET / HTTP/1.1
   > Host: www.example.com
   > User-Agent: curl/8.1.2
   > Accept: */*
   >

   Use the -v or --verbose parameter to display detailed information about the request and the server's response, and 2>&1 | grep '^>' parameter to filter the output. These are for demonstration purposes only and are not required for normal cURL requests.

3. Change the User-Agent string using the -A option.

   $ curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" http://www.example.com -v 2>&1 | grep '^>'
   > GET / HTTP/1.1
   > Host: www.example.com
   > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
   > Accept: */*

   Related: List of Browser User Agents

4. Use the longer %–%user-agent option for the same result.

   $ curl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" http://www.example.com -v 2>&1 | grep '^>'
   > GET / HTTP/1.1
   > Host: www.example.com
   > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
   > Accept: */*

5. Manually set the User-Agent header using the -H option.

   $ curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" http://www.example.com -v 2>&1 | grep '^>'
   > GET / HTTP/1.1
   > Host: www.example.com
   > Accept: */*
   > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

   Related: How to set custom headers in cURL requests

6. Check the server’s response to your request by sending a HEAD request using -I.

   $ curl -I www.example.com
   HTTP/1.1 200 OK
   Server: nginx
   Date: Sat, 30 Sep 2023 04:28:05 GMT
   Content-Type: text/html
   Content-Length: 1991
   Connection: keep-alive
   Keep-Alive: timeout=20
   Vary: Accept-Encoding
   Last-Modified: Thu, 26 Aug 2021 07:13:07 GMT
   ETag: "61273f03-7c7"
   Accept-Ranges: bytes

   The -I flag sends a HEAD request, which returns only the headers, helping observe potential variations in server responses based on the User-Agent.

7. Permanently set the User-Agent in the cURL configuration file.

   $ echo 'user-agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"' >> ~/.curlrc

   Encapsulate the User-Agent string in double quotes and the entire command in single quotes.

8. Make a new request to verify the updated User-Agent without specifying it explicitly.

   $ curl http://www.example.com -v 2>&1 | grep '^>'
   > GET / HTTP/1.1
   > Host: www.example.com
   > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
   > Accept: */*