How to configure retries in Scrapy

Retries keep a Scrapy crawl moving through temporary failures such as overloaded upstreams, rate limits, and short network interruptions without forcing a full spider rerun.

Scrapy handles retries in RetryMiddleware, which watches configured HTTP status codes and selected download exceptions, increments the request's retry_times metadata, and reschedules the failed request. Failed requests are queued behind the regular crawl work, so retries usually run after the first pass through the pending request queue.

Current Scrapy releases enable retries by default and use RETRY_TIMES = 2 unless the project changes it. Raising the retry budget too far increases crawl time and request volume, so keep the count modest, avoid retrying permanent failures such as 404 unless there is a clear reason, and use request-level overrides when only one request needs different retry behavior.

1. Open the Scrapy project settings file.

   $ vi quotesbot/settings.py

   In a default project layout, the file is usually <project_name>/settings.py.

2. Set the project-wide retry policy in settings.py.

   RETRY_ENABLED = True
   RETRY_TIMES = 3
   RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]

   Current Scrapy defaults already enable retries, retry these eight response codes, and use 2 retries per request. Keeping the values explicit in settings.py makes the project policy easier to review and tune.

3. Confirm the project is loading the intended retry count.

   $ scrapy settings --get RETRY_TIMES
   3

   Run scrapy settings –get RETRY_HTTP_CODES from the project root when the active retryable status-code list also needs to be checked.

4. Override the retry budget on a single request when one endpoint needs a different limit.

   yield scrapy.Request(
       detail_url,
       callback=self.parse_detail,
       meta={"max_retry_times": 5},
   )

   max_retry_times overrides RETRY_TIMES for one request only. Set dont_retry to True instead when a request should fail fast.

5. Run the spider with DEBUG logging to watch retry activity.

   $ scrapy crawl products -s LOG_LEVEL=DEBUG
   2026-04-16 15:02:11 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://catalog.example.com/products/retry-demo> (failed 1 times): 503 Service Unavailable
   2026-04-16 15:02:11 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://catalog.example.com/products/retry-demo> (failed 2 times): 503 Service Unavailable
   2026-04-16 15:02:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://catalog.example.com/products/retry-demo> (referer: None)
   2026-04-16 15:02:11 [scrapy.core.scraper] DEBUG: Scraped from <200 https://catalog.example.com/products/retry-demo>
   {'status': 200, 'title': 'Retry demo success'}

   Retry messages are logged at DEBUG level, so long crawls can generate large log files and expose full request URLs in the log stream.

6. Check the crawl stats for retry counters after the spider finishes.

   2026-04-16 15:02:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
   {'downloader/request_count': 3,
    'downloader/response_status_count/200': 1,
    'downloader/response_status_count/503': 2,
    'item_scraped_count': 1,
    'retry/count': 2,
    'retry/reason_count/503 Service Unavailable': 2,
   ##### snipped #####
   }

   retry/count reports extra attempts beyond the first download. A value of 2 means the request ran three total times before it succeeded.