How to set the log level in Scrapy

Adjusting the Scrapy log level keeps crawler output readable while still surfacing meaningful failures quickly during long-running spiders and automated runs.

Scrapy writes messages through Python’s logging system and filters them using the LOG_LEVEL setting. The value is commonly set in the project settings module, with optional overrides available per spider via custom_settings or per run using the -s command-line option.

Verbose levels such as DEBUG can include request URLs, headers, cookies, redirects, and retry details that may leak tokens or grow log files quickly. Quieter levels such as WARNING reduce noise but can hide useful progress information, so routine crawls typically prefer INFO or WARNING while short troubleshooting runs temporarily switch to DEBUG.

1. Open the Scrapy project settings file.

   $ vi simplifiedguide/settings.py

   The settings module is typically located at simplifiedguide/settings.py inside the Scrapy project directory.

2. Set the LOG_LEVEL value to the desired level.

   LOG_LEVEL = "INFO"

   Common levels include CRITICAL, ERROR, WARNING, INFO, and DEBUG.

   A one-off override can be applied without editing files, such as scrapy crawl products -s LOG_LEVEL=DEBUG.

   DEBUG output can expose cookies, headers, and tokens in terminal scrollback and log files.

3. Run the spider to confirm the updated log level.

   $ scrapy crawl products
   2026-01-01 08:26:24 [scrapy.utils.log] INFO: Scrapy 2.11.1 started (bot: simplifiedguide)
   2026-01-01 08:26:24 [scrapy.utils.log] INFO: Versions: lxml 5.2.1.0, libxml2 2.9.14, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0], pyOpenSSL 23.2.0 (OpenSSL 3.0.13 30 Jan 2024), cryptography 41.0.7, Platform Linux-6.12.54-linuxkit-aarch64-with-glibc2.39
   ##### snipped #####

   Setting WARNING suppresses the INFO startup banner and routine crawl stats while still showing ERROR and CRITICAL messages.