How to set a download delay in Scrapy

Setting a download delay in Scrapy slows back-to-back requests to the same target so crawls stay closer to site limits, reduce burst traffic, and look less like abusive automation.

Scrapy applies DOWNLOAD_DELAY per downloader slot, which is usually per domain, as the minimum wait between consecutive requests for that slot. Decimal values are supported, and the setting works with concurrency limits and AutoThrottle rather than replacing them.

Current projects created with scrapy startproject already include DOWNLOAD_DELAY = 1 and CONCURRENT_REQUESTS_PER_DOMAIN = 1 in settings.py, so changing the existing line is usually cleaner than adding a duplicate setting. RANDOMIZE_DOWNLOAD_DELAY is still enabled by default, and when CONCURRENT_REQUESTS_PER_IP is non-zero Scrapy enforces the delay per IP instead of per domain.

1. Open the Scrapy project settings file.

   $ vi simplifiedguide/settings.py

   In a standard project layout, the file is usually <project_name>/settings.py.

2. Change the existing DOWNLOAD_DELAY line to the delay value that the crawl should enforce.

   DOWNLOAD_DELAY = 2.0

   Decimal seconds are supported, so values such as 0.5, 1.5, and 2.5 are valid.

3. Check the effective delay and delay-randomization values that the project is loading.

   $ scrapy settings --get DOWNLOAD_DELAY
   2.0
   $ scrapy settings --get RANDOMIZE_DOWNLOAD_DELAY
   True

   With randomization enabled, Scrapy waits between roughly half and one and a half times DOWNLOAD_DELAY for the same slot.

4. Run the spider and watch the request timestamps to confirm the same slot is being paced at the configured interval.

   $ scrapy crawl products -s RANDOMIZE_DOWNLOAD_DELAY=False
   2026-04-16 05:48:05 [scrapy.crawler] INFO: Overridden settings:
   {'DOWNLOAD_DELAY': 2.0,
    'RANDOMIZE_DOWNLOAD_DELAY': 'False'}
   ##### snipped #####
   2026-04-16 05:48:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://catalog.example/products/> (referer: None)
   2026-04-16 05:48:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://catalog.example/products/page-2/> (referer: http://catalog.example/products/)
   2026-04-16 05:48:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://catalog.example/products/page-3/> (referer: http://catalog.example/products/page-2/)

   Disabling RANDOMIZE_DOWNLOAD_DELAY for this run makes the interval easier to see in the log. Restore the default randomized behavior for normal crawls unless a fixed pause is required.

   Spider-level settings such as download_delay or custom_settings, and higher parallelism from CONCURRENT_REQUESTS_PER_DOMAIN, can change the observed timing even when the project setting is correct.