How to block user agents in Apache

Blocking abusive User-Agent strings in Apache reduces log noise from commodity scanners, lowers wasted CPU on expensive routes, and keeps a single misbehaving bot from turning normal traffic into a slow-motion denial of service.

Apache exposes the User-Agent header to configuration directives during request processing. Using SetEnvIfNoCase from mod_setenvif, matching requests can be tagged with an internal environment variable and then filtered by Apache 2.4 authorization rules.

Because the User-Agent header is client-supplied, spoofing is trivial; treat blocking as a coarse filter, not a security boundary. Keep match patterns narrow, scope rules to the correct content directory for the site, and run a syntax test before reloading so a bad directive does not prevent Apache from applying changes.

1. Enable the setenvif module when it is not already loaded.

   $ sudo a2enmod setenvif

   Skip this step on distros that do not use a2enmod or where the module is already enabled.

2. Create /etc/apache2/conf-available/block-user-agent.conf with a SetEnvIfNoCase rule that marks unwanted clients.

   $ sudo tee /etc/apache2/conf-available/block-user-agent.conf >/dev/null <<'EOF'
   SetEnvIfNoCase User-Agent "badbot|scanner|crawler" bad_ua=1
   EOF

   Use specific patterns to avoid blocking legitimate crawlers, uptime checks, or real browsers.

3. Add an access rule that denies requests when bad_ua is set.

   $ sudo tee -a /etc/apache2/conf-available/block-user-agent.conf >/dev/null <<'EOF'
   <Directory "/var/www/">
       <RequireAll>
           Require all granted
           Require not env bad_ua
       </RequireAll>
   </Directory>
   EOF

   Change /var/www/ to match the DocumentRoot path used by the site when content is served from a different directory.

4. Enable the configuration snippet.

   $ sudo a2enconf block-user-agent

5. Validate configuration syntax before reloading.

   $ sudo apachectl configtest
   Syntax OK

   Related: How to test your Apache configuration

6. Reload Apache to apply the change without dropping existing connections.

   $ sudo systemctl reload apache2

   Rollback: disable the snippet with

   $ sudo a2disconf block-user-agent

   and reload again.

   Related: How to manage the Apache web server service

7. Verify the block rule by sending a request with a matching user agent.

   $ curl -i -A 'badbot' http://127.0.0.1/
   HTTP/1.1 403 Forbidden
   ##### snipped #####

8. Confirm normal requests are still allowed with a non-matching user agent.

   $ curl -I http://127.0.0.1/
   HTTP/1.1 200 OK
   ##### snipped #####