How to set a download timeout in Scrapy

Setting a download timeout in Scrapy keeps stalled responses from occupying downloader slots indefinitely. A shorter timeout helps crawls recover faster from dead or overloaded endpoints, while a longer timeout leaves more room for large responses and slower upstream services to finish.

Scrapy applies the limit through DownloadTimeoutMiddleware and the DOWNLOAD_TIMEOUT setting. When the timer is exceeded, the request fails with DownloadTimeoutError, which then feeds into retry logic, errbacks, and crawl statistics like any other download failure.

The project-wide default is 180 seconds. Current Scrapy releases also support request-level download_timeout in Request.meta and per-spider overrides through custom_settings, while the older download_timeout spider attribute is deprecated and is no longer the strongest pattern for new spiders.

1. Open the Scrapy project settings file.

   $ vi simplifiedguide/settings.py

   In a default project layout, the file is usually <project_name>/settings.py.

2. Set DOWNLOAD_TIMEOUT to the preferred limit in seconds.

   DOWNLOAD_TIMEOUT = 20

   Low values can turn normally slow responses into timeout failures, while high values hold downloader slots longer and reduce crawl throughput on bad connections.

3. Add a spider-specific override through custom_settings when only one spider needs a different timeout.

   import scrapy
    
   class ExampleSpider(scrapy.Spider):
       name = "example"
       custom_settings = {
           "DOWNLOAD_TIMEOUT": 30,
       }

   Use custom_settings for per-spider overrides in current Scrapy instead of the deprecated download_timeout spider attribute.

   Related: How to use custom settings in Scrapy

4. Set download_timeout in Request.meta when only one request needs a different limit.

   yield scrapy.Request(
       url,
       meta={"download_timeout": 10},
   )

   A request-level override is useful for unusually slow endpoints such as exports, search pages, or paginated APIs without changing the rest of the spider.

5. Confirm the project default timeout from the project root.

   $ scrapy settings --get DOWNLOAD_TIMEOUT
   20

   This command shows the active project default and does not include per-spider or per-request overrides.

6. Run the spider and confirm the crawl starts with the expected timeout when custom_settings is used.

   $ scrapy crawl example
   2026-04-16 13:49:04 [scrapy.crawler] INFO: Overridden settings:
   {'DOWNLOAD_TIMEOUT': 30,
   ##### snipped #####
   }

   Per-request download_timeout overrides are applied at runtime, so they do not appear in the startup banner.

   Slow responses that exceed the active limit fail as DownloadTimeoutError in the crawl log.