How to enable AutoThrottle in Scrapy

AutoThrottle keeps a Scrapy crawl responsive when a target site speeds up or slows down, which helps reduce accidental overload, retry storms, and rate-limit responses during longer runs.

Scrapy assigns requests to download slots, usually by domain, and the AutoThrottle extension adjusts each slot delay from observed response latency. The extension tries to approach the configured AUTOTHROTTLE_TARGET_CONCURRENCY while still respecting the normal DOWNLOAD_DELAY floor and the active CONCURRENT_REQUESTS_PER_DOMAIN limit.

In current Scrapy releases, AUTOTHROTTLE_ENABLED is disabled by default, AUTOTHROTTLE_START_DELAY defaults to 5.0 seconds, AUTOTHROTTLE_MAX_DELAY defaults to 60.0 seconds, and AUTOTHROTTLE_TARGET_CONCURRENCY defaults to 1.0. Lowering the starting delay makes the first requests ramp up sooner, raising the target concurrency increases pressure on the remote site, and AUTOTHROTTLE_DEBUG adds one log line for every response.

1. Open the Scrapy project settings file.

   $ vi simplifiedguide/settings.py

   In a default project layout, the file is usually <project_name>/settings.py.

2. Enable AutoThrottle and set the initial pacing values for the project.

   AUTOTHROTTLE_ENABLED = True
   AUTOTHROTTLE_START_DELAY = 1.0
   AUTOTHROTTLE_MAX_DELAY = 60.0
   AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

   The example keeps the current AUTOTHROTTLE_MAX_DELAY and AUTOTHROTTLE_TARGET_CONCURRENCY defaults, but lowers the initial delay from 5.0 to 1.0 so responsive targets ramp up sooner.

   AutoThrottle adjusts delay, not hard concurrency, and it never drives the slot below DOWNLOAD_DELAY or above the current per-domain concurrency ceiling.

3. Check the effective AutoThrottle values that the project is loading before the next crawl.

   $ scrapy settings --get AUTOTHROTTLE_ENABLED
   True
   $ scrapy settings --get AUTOTHROTTLE_START_DELAY
   1.0
   $ scrapy settings --get AUTOTHROTTLE_TARGET_CONCURRENCY
   1.0
   $ scrapy settings --get DOWNLOAD_DELAY
   1

   If DOWNLOAD_DELAY prints 1, the first AutoThrottle debug lines may stay at 1000 ms until slower responses push the delay higher.

4. Run the spider with AutoThrottle debug mode enabled and watch the per-slot delay lines in the log.

   $ scrapy crawl products -s AUTOTHROTTLE_DEBUG=True -s LOG_LEVEL=INFO
   2026-04-16 05:31:45 [scrapy.utils.log] INFO: Scrapy 2.15.0 started (bot: simplifiedguide)
   2026-04-16 05:31:45 [scrapy.crawler] INFO: Overridden settings:
   {'AUTOTHROTTLE_DEBUG': 'True',
    'AUTOTHROTTLE_ENABLED': True,
    'AUTOTHROTTLE_START_DELAY': 1.0,
    'BOT_NAME': 'simplifiedguide',
    'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
    'DOWNLOAD_DELAY': 1,
    'LOG_LEVEL': 'INFO'}
   ##### snipped #####
   2026-04-16 05:31:46 [scrapy.extensions.throttle] INFO: slot: catalog.example | conc: 1 | delay: 1000 ms (+0) | latency:    8 ms | size:   460 bytes
   2026-04-16 05:31:47 [scrapy.extensions.throttle] INFO: slot: catalog.example | conc: 1 | delay: 1000 ms (+0) | latency:    6 ms | size:    91 bytes
   2026-04-16 05:31:48 [scrapy.extensions.throttle] INFO: slot: catalog.example | conc: 1 | delay: 1000 ms (+0) | latency:    1 ms | size:    40 bytes

   Seeing scrapy.extensions.throttle lines is the decisive success check. The exact slot value follows the domain or IP being crawled, and a steady 1000 ms line can still be correct when DOWNLOAD_DELAY is set to 1.

   Keep AUTOTHROTTLE_DEBUG as a run-time override for troubleshooting because it writes a new throttle line for every response.