How to use Scrapy shell

Testing extraction logic in Scrapy before committing it to a spider reduces failed crawls, wasted requests, and brittle parsing code. The interactive shell makes it practical to confirm selectors and field cleanup rules quickly, especially when target pages change frequently.

Running scrapy shell downloads a single URL through the same downloader, middleware, and project settings used during a normal crawl. The session exposes a Response object as response with css() and xpath() helpers, making it straightforward to iterate on selectors and reuse the working expressions inside a spider’s parse() callback.

The shell does not execute JavaScript, so pages that build content client-side may appear empty or incomplete even when the request succeeds. Anti-bot rules, redirects, rate limits, cookies, and custom headers still apply, so keep experiments lightweight and use per-run -s NAME=VALUE overrides when a site requires a specific User-Agent or cookie behavior.

1. Open a terminal in the Scrapy project directory.

   $ cd /root/sg-work/simplifiedguide

2. Start the shell with the target URL.

   $ scrapy shell http://app.internal.example:8000/products/
   2026-01-01 06:33:52 [scrapy.utils.log] INFO: Scrapy 2.11.1 started (bot: simplifiedguide)
   2026-01-01 06:33:52 [scrapy.utils.log] INFO: Versions: lxml 5.2.1.0, libxml2 2.9.14, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0], pyOpenSSL 23.2.0 (OpenSSL 3.0.13 30 Jan 2024), cryptography 41.0.7, Platform Linux-6.12.54-linuxkit-aarch64-with-glibc2.39
   ##### snipped #####

   Override settings for a single run with -s NAME=VALUE , such as -s USER_AGENT="Mozilla/5.0 (X11; Linux x86_64)" when a site blocks the default Scrapy identifier.

3. Check the HTTP status code from the response.

   In [1]: response.status
   Out[1]: 200

   Status codes other than 200 commonly indicate redirects, blocks, or missing pages.

4. Check the final response URL after redirects.

   In [2]: response.url
   Out[2]: 'http://app.internal.example:8000/products/'

5. Confirm expected server-rendered content exists in the HTML.

   In [3]: response.css('title::text').get()
   Out[3]: 'Products - Example Store'

   Empty or placeholder HTML commonly indicates a JavaScript-rendered page.

6. List values from a selector that should match multiple items.

   In [4]: response.css('article.product h2::text').getall()
   Out[4]: ['Starter Plan', 'Team Plan', 'Enterprise Plan']

   XPath selectors can be tested with response.xpath() using the same response.

7. Extract and clean a single field value for one item.

   In [5]: response.css('article.product span.price::text').get().strip()
   Out[5]: '$29'

8. Fetch another page URL without leaving the shell to test pagination or detail pages.

   In [6]: fetch('http://app.internal.example:8000/products?page=2')
   In [7]: response.url
   Out[7]: 'http://app.internal.example:8000/products?page=2'

   Each fetch() triggers a real request and can trip rate limits or anti-bot rules when repeated aggressively.

9. Copy the verified selectors into the spider parse() method.

   import scrapy
   
   class ProductsSpider(scrapy.Spider):
       name = "products"
       start_urls = ["http://app.internal.example:8000/products/"]
   
       def parse(self, response):
           for product in response.css("article.product"):
               yield {
                   "name": product.css("h2::text").get(),
                   "price": product.css("span.price::text").get(),
               }

   Related: How to create a Scrapy spider