How to change user agent for Selenium spiders

Server rules, analytics logs, and crawl allowlists often branch on the HTTP User-Agent header. A Selenium spider that keeps Chrome's default browser identity can be sorted into the wrong path, so set the crawler's agreed browser-shaped string before the first page request.

ChromeDriver receives Chrome startup arguments through Selenium ChromeOptions. Adding --user-agent before webdriver.Chrome() creates the session makes both the HTTP request header and navigator.userAgent report the custom value for pages loaded in that browser.

Changing the string does not make Chrome behave like another device, and modern sites can also inspect User-Agent Client Hints, screen size, cookies, IP reputation, or automation signals. Use a truthful spider token for targets that allow crawling, and verify the value against a request-inspection endpoint before pointing the run at a production site.

1. Choose the exact user-agent string for the spider.

   Keep a browser-shaped base string when the target expects a normal Chrome request, and add a crawler token such as SeleniumSpider/1.0 when the target's policy asks crawlers to identify themselves.

2. Create a smoke-test spider that adds the user-agent argument before starting Chrome.

   $ cat > selenium-change-user-agent.py <<'PY'
   from selenium import webdriver
   from selenium.webdriver.chrome.options import Options
   from selenium.webdriver.common.by import By
   
   user_agent = (
       "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
       "(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36 "
       "SeleniumSpider/1.0"
   )
   
   options = Options()
   options.add_argument("--headless=new")
   options.add_argument(f"--user-agent={user_agent}")
   
   driver = webdriver.Chrome(options=options)
   try:
       driver.get("https://httpbin.org/user-agent")
       print(driver.find_element(By.TAG_NAME, "body").text)
       print(f"navigator.userAgent: {driver.execute_script('return navigator.userAgent')}")
   finally:
       driver.quit()
   PY

   The --user-agent argument belongs on the Options object before the driver is created. Set it before the first request so the initial navigation uses the custom header.

3. Run the smoke-test spider.

   $ python3 selenium-change-user-agent.py
   {
     "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36 SeleniumSpider/1.0"
   }
   navigator.userAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36 SeleniumSpider/1.0

4. Copy the same Options block into the real spider before changing the target URL.

   Do not use a random or misleading user-agent rotation to bypass a site's crawl rules. User-agent strings are commonly logged with rate limits, allowlists, and abuse investigations.

5. Remove the temporary smoke-test file after the spider code is updated.

   $ rm selenium-change-user-agent.py