How to set a download delay in Scrapy

Setting a download delay in Scrapy reduces pressure on target sites and helps crawls stay under rate limits. Predictable pacing is less likely to trigger automated blocks and keeps dynamic pages from getting hammered by bursts of concurrent requests. A sensible delay also makes crawler traffic easier to defend when policies or abuse contacts get involved.

Scrapy schedules requests concurrently and assigns them to downloader “slots” (typically per domain). The DOWNLOAD_DELAY setting adds a pause before starting the next request for the same slot, shaping request cadence without changing spider code. It works alongside concurrency controls so per-domain throughput can be tuned with delay and parallelism together.

Large delays can make crawls impractically slow, while small delays can still overload smaller sites when concurrency is high. RANDOMIZE_DOWNLOAD_DELAY is enabled by default, so the actual pause varies around the configured value to avoid fixed timing patterns. For adaptive pacing based on latency and server responsiveness, AutoThrottle can be enabled instead of relying only on a fixed delay.

1. Open the Scrapy project settings file.

   $ vi simplifiedguide/settings.py

   In a default project layout, the file is usually <project_name>/settings.py.

2. Set DOWNLOAD_DELAY to the delay value in seconds.

   DOWNLOAD_DELAY = 2.0

   With RANDOMIZE_DOWNLOAD_DELAY enabled (default), a value of 2.0 is randomized to roughly 1.0 to 3.0 seconds per request slot.

3. Disable RANDOMIZE_DOWNLOAD_DELAY when an exact fixed pause is required.

   RANDOMIZE_DOWNLOAD_DELAY = False

   Fixed timing is easier to measure, but highly predictable intervals can be easier to fingerprint.

4. Run the spider to produce request timestamps in the log.

   $ scrapy crawl products
   2026-01-01 08:19:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://app.internal.example:8000/products/> (referer: None)
   2026-01-01 08:19:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://app.internal.example:8000/products?page=2> (referer: http://app.internal.example:8000/products/)
   ##### snipped #####

   The timestamp gap also includes network and parsing time, so consistency is easier to observe with simple pages and low concurrency.