How to enable AutoThrottle in Scrapy

Enabling AutoThrottle in Scrapy lets a crawl slow down when a target site responds slowly and speed up again when the target is keeping up. That reduces burst traffic, retry pressure, and avoidable rate-limit responses during longer runs.

Scrapy adjusts the delay for each download slot, which is usually one slot per domain, from measured response latency. The extension aims for the configured AUTOTHROTTLE_TARGET_CONCURRENCY, but it still respects the active DOWNLOAD_DELAY floor and the CONCURRENT_REQUESTS_PER_DOMAIN limit.

Current projects created with scrapy startproject already include a commented AutoThrottle block in settings.py, with AUTOTHROTTLE_ENABLED off by default, AUTOTHROTTLE_START_DELAY at 5 seconds, AUTOTHROTTLE_MAX_DELAY at 60 seconds, and AUTOTHROTTLE_TARGET_CONCURRENCY at 1.0. If an older project still sets CONCURRENT_REQUESTS_PER_IP, the practical limit moves from per-domain to per-IP, and enabling AUTOTHROTTLE_DEBUG adds one log line for every response.

1. Open the Scrapy project settings file and locate the commented AutoThrottle block.

   $ vi simplifiedguide/settings.py

   In a default project layout, the file is usually <project_name>/settings.py, and new projects already include the AutoThrottle lines as commented template entries.

2. Uncomment or add the AutoThrottle settings and set the starting delay that the crawl should use.

   AUTOTHROTTLE_ENABLED = True
   AUTOTHROTTLE_START_DELAY = 1.0
   AUTOTHROTTLE_MAX_DELAY = 60.0
   AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

   The example keeps the current AUTOTHROTTLE_MAX_DELAY and AUTOTHROTTLE_TARGET_CONCURRENCY defaults, but lowers the initial delay from 5.0 to 1.0 so responsive targets ramp up sooner.

   AutoThrottle adjusts delay, not hard concurrency, and it never drives the slot below DOWNLOAD_DELAY or above the current per-domain concurrency ceiling.

3. Check that the project is loading AUTOTHROTTLE_ENABLED before the next crawl.

   $ scrapy settings --get AUTOTHROTTLE_ENABLED
   True

   Run the same command with AUTOTHROTTLE_START_DELAY, AUTOTHROTTLE_MAX_DELAY, or AUTOTHROTTLE_TARGET_CONCURRENCY if one of those values was changed from the example.

4. Run the spider with AutoThrottle debug mode enabled and watch the per-slot delay lines in the log.

   $ scrapy crawl products -s AUTOTHROTTLE_DEBUG=True -s LOG_LEVEL=INFO
   2026-04-22 05:49:06 [scrapy.crawler] INFO: Overridden settings:
   {'AUTOTHROTTLE_DEBUG': 'True',
    'AUTOTHROTTLE_ENABLED': True,
    'AUTOTHROTTLE_START_DELAY': 1.0,
    'BOT_NAME': 'simplifiedguide',
    'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
    'DOWNLOAD_DELAY': 1,
    'LOG_LEVEL': 'INFO'}
   ##### snipped #####
   2026-04-22 05:49:08 [scrapy.extensions.throttle] INFO: slot: 127.0.0.1 | conc: 1 | delay: 1000 ms (+0) | latency:  139 ms | size:    33 bytes
   2026-04-22 05:49:09 [scrapy.extensions.throttle] INFO: slot: 127.0.0.1 | conc: 1 | delay: 1000 ms (+0) | latency:  208 ms | size:   103 bytes
   2026-04-22 05:49:11 [scrapy.extensions.throttle] INFO: slot: 127.0.0.1 | conc: 1 | delay: 1000 ms (+0) | latency:  614 ms | size:    31 bytes

   Seeing scrapy.extensions.throttle lines proves the extension is active, and a steady 1000 ms delay is expected when DOWNLOAD_DELAY is still 1.

   Leave AUTOTHROTTLE_DEBUG as a run-time override because it logs one extra line for every response.