How to change user agent for Selenium spiders

Web servers often separate crawler traffic by the identity supplied with each request. A Selenium spider can send a clear, site-approved identity by setting Chrome's User-Agent value before the browser session starts.

Chrome receives startup arguments from Selenium's Options object. The --user-agent argument changes both the classic HTTP request header and navigator.userAgent for pages opened by that browser session.

A user-agent string is a self-reported label rather than proof of browser type or crawler ownership. Use a truthful spider token with a contact URL, follow the target site's crawl policy, and treat Client Hints, viewport size, IP address, and automation signals as separate properties.

Steps to change Selenium spider user agent:

Choose a truthful crawler identity with a contact URL for the User-Agent header.

The value SeleniumSpider/1.0 (+https://www.example.com/bot) identifies the crawler without claiming a specific Chrome release.
Tool: User-Agent Parser

Create selenium-change-user-agent.py with the crawler identity and Chrome startup options.

$ cat > selenium-change-user-agent.py <<'PY'
import json

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

user_agent = "SeleniumSpider/1.0 (+https://www.example.com/bot)"

options = Options()
options.add_argument("--headless=new")
options.add_argument(f"--user-agent={user_agent}")
PY

The Options object must receive --user-agent before webdriver.Chrome() creates the browser session.

Append the request and fail-capable identity checks to selenium-change-user-agent.py.

$ cat >> selenium-change-user-agent.py <<'PY'

with webdriver.Chrome(options=options) as driver:
    driver.get("https://httpbin.org/user-agent")
    header = json.loads(driver.find_element(By.TAG_NAME, "body").text)["user-agent"]
    browser = driver.execute_script("return navigator.userAgent")

if header != user_agent or browser != user_agent:
    raise RuntimeError(
        f"user agent mismatch: header={header!r}, browser={browser!r}"
    )

print(f"HTTP User-Agent: {header}")
print(f"navigator.userAgent: {browser}")
PY

The request endpoint returns the HTTP header received by the server, while the script reads navigator.userAgent from the same browser session.

Run the completed spider to confirm the HTTP header and browser property match.
```
$ python3 selenium-change-user-agent.py
HTTP User-Agent: SeleniumSpider/1.0 (+https://www.example.com/bot)
navigator.userAgent: SeleniumSpider/1.0 (+https://www.example.com/bot)
```
Misleading identity rotation can violate crawl restrictions or rate limits, and sites can correlate the string with request volume, addresses, cookies, and other automation signals.