How to configure feed exports in Scrapy

Feed exports keep scraped item output consistent across runs by moving file naming, format selection, and encoding into the project configuration instead of ad-hoc command flags. Predictable output makes it easier to test spiders, compare crawl results, and feed downstream jobs without manual cleanup.

Scrapy’s feed exporter reads the FEEDS setting to determine where items are written and how they are serialized. Each entry maps a destination URI (local path, stdout:, or a supported storage backend) to exporter options such as format and encoding, and multiple destinations can be written from the same crawl.

Relative file paths in FEEDS are resolved from the current working directory used to start the crawl, so running from the project directory keeps output locations consistent. Local file targets do not create missing directories, and enabling overwrite replaces existing files at that path, so choose a dedicated output folder to avoid accidental data loss.

1. Change to the Scrapy project directory that contains scrapy.cfg.

   $ cd /root/sg-work/catalog_demo

   Relative feed paths in FEEDS resolve from the current working directory.

2. Open the Scrapy settings file for the project.

   $ vi catalog_demo/settings.py

3. Define the FEEDS setting with the desired output path and exporter options.

   FEEDS = {
       "output/products.jsonl": {
           "format": "jsonlines",
           "encoding": "utf8",
           "overwrite": True,
       },
   }

   Enabling overwrite replaces any existing file at the same path.

   URI placeholders such as %(name)s and %(time)s can generate per-spider and per-run filenames.

4. Create the destination directory referenced by the feed path.

   $ mkdir -p output

   Missing directories cause local file exports to fail.

5. Run the spider to write the feed output.

   $ scrapy crawl catalog
   2026-01-01 09:39:32 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (6 items) in: output/products.jsonl

   The feed storage log line appears when the crawl finishes.

6. Count exported records to confirm one line was written per item.

   $ wc -l output/products.jsonl
   6 output/products.jsonl

7. Preview the first few records to confirm jsonlines output structure.

   $ head -n 3 output/products.jsonl
   {"name": "Starter Plan", "price": "$29", "url": "http://app.internal.example:8000/products/starter-plan.html"}
   {"name": "Team Plan", "price": "$79", "url": "http://app.internal.example:8000/products/team-plan.html"}
   {"name": "Enterprise Plan", "price": "$199", "url": "http://app.internal.example:8000/products/enterprise-plan.html"}