How to configure retries in Scrapy

Retries keep a Scrapy crawl moving through short-lived network failures and temporary server problems, reducing gaps that would otherwise require re-running spiders or backfilling missed URLs.

Scrapy applies retries through the downloader middleware stack, where RetryMiddleware detects retryable HTTP responses and download exceptions, increments a per-request retry_times counter, and schedules the request again with an adjusted priority.

Every retry increases request volume and crawl duration, so keep counts modest and treat retryable codes as signals to slow down rather than to hammer harder. Avoid retrying non-idempotent actions (for example POST requests that create records or trigger side effects) by skipping retries per request, and be cautious when adding permanent errors like 404 or access bans like 403 to the retry list.

1. Open the Scrapy project settings file (for example, simplifiedguide/settings.py).

   $ vi simplifiedguide/settings.py

2. Set the retry policy in the settings file.

   RETRY_ENABLED = True
   RETRY_TIMES = 3
   RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]

   RETRY_TIMES counts retries per request, so total attempts is 1 + RETRY_TIMES. Retries are skipped when dont_retry is set in Request.meta.

3. Run the spider with DEBUG logging enabled to observe retry activity.

   $ scrapy crawl products -s LOG_LEVEL=DEBUG

   Using DEBUG logging can generate large log files and may expose request URLs and headers in logs.

4. Confirm RetryMiddleware emits retry messages for retryable failures.

   2026-01-01 08:23:31 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://app.internal.example:8000/errors/503> (failed 1 times): 503 Service Unavailable
   2026-01-01 08:23:33 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://app.internal.example:8000/errors/503> (failed 2 times): 503 Service Unavailable
   2026-01-01 08:23:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://app.internal.example:8000/errors/503> (failed 3 times): 503 Service Unavailable
   2026-01-01 08:23:37 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET http://app.internal.example:8000/errors/503> (failed 4 times): 503 Service Unavailable
   ##### snipped #####

5. Check the crawl summary for retry counters.

   2026-01-01 08:23:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
   {'downloader/request_count': 5,
    'downloader/response_status_count/200': 1,
    'downloader/response_status_count/503': 4,
    'retry/count': 3,
    'retry/reason_count/503 Service Unavailable': 3,
   ##### snipped #####
   }