How to change user agent for Selenium spiders

Web servers often separate crawler traffic by the identity supplied with each request. A Selenium spider can send a clear, site-approved identity by setting Chrome's User-Agent value before the browser session starts.

Chrome receives startup arguments from Selenium's Options object. The --user-agent argument changes both the classic HTTP request header and navigator.userAgent for pages opened by that browser session.

A user-agent string is a self-reported label rather than proof of browser type or crawler ownership. Use a truthful spider token with a contact URL, follow the target site's crawl policy, and treat Client Hints, viewport size, IP address, and automation signals as separate properties.

1. Choose a truthful crawler identity with a contact URL for the User-Agent header.

   The value SeleniumSpider/1.0 (+https://www.example.com/bot) identifies the crawler without claiming a specific Chrome release.
   Tool: User-Agent Parser

2. Create selenium-change-user-agent.py with the crawler identity and Chrome startup options.

   $ cat > selenium-change-user-agent.py <<'PY'
   import json
   
   from selenium import webdriver
   from selenium.webdriver.common.by import By
   from selenium.webdriver.chrome.options import Options
   
   user_agent = "SeleniumSpider/1.0 (+https://www.example.com/bot)"
   
   options = Options()
   options.add_argument("--headless=new")
   options.add_argument(f"--user-agent={user_agent}")
   PY

   The Options object must receive --user-agent before webdriver.Chrome() creates the browser session.

3. Append the request and fail-capable identity checks to selenium-change-user-agent.py.

   $ cat >> selenium-change-user-agent.py <<'PY'
   
   with webdriver.Chrome(options=options) as driver:
       driver.get("https://httpbin.org/user-agent")
       header = json.loads(driver.find_element(By.TAG_NAME, "body").text)["user-agent"]
       browser = driver.execute_script("return navigator.userAgent")
   
   if header != user_agent or browser != user_agent:
       raise RuntimeError(
           f"user agent mismatch: header={header!r}, browser={browser!r}"
       )
   
   print(f"HTTP User-Agent: {header}")
   print(f"navigator.userAgent: {browser}")
   PY

   The request endpoint returns the HTTP header received by the server, while the script reads navigator.userAgent from the same browser session.

4. Run the completed spider to confirm the HTTP header and browser property match.

   $ python3 selenium-change-user-agent.py
   HTTP User-Agent: SeleniumSpider/1.0 (+https://www.example.com/bot)
   navigator.userAgent: SeleniumSpider/1.0 (+https://www.example.com/bot)

   Misleading identity rotation can violate crawl restrictions or rate limits, and sites can correlate the string with request volume, addresses, cookies, and other automation signals.