How to configure feed exports in Scrapy

Configuring feed exports in Scrapy moves the output path, serialization format, and field order into project settings so repeated crawls produce the same export layout every time. That keeps downstream processing predictable and removes the need to remember ad-hoc output flags for each spider run.

Scrapy reads the FEEDS setting as a dictionary of destination URIs mapped to feed-specific options. Each feed definition can set the export format, exported fields, overwrite behavior, and other feed options, while the output URI itself selects the storage backend.

Current Scrapy releases still treat local filesystem feeds as append targets unless overwrite is set to True, and startproject now seeds FEED_EXPORT_ENCODING with utf-8 in new projects. Feed URIs can also use placeholders such as %(name)s and %(time)s when each spider or crawl run should write a separate file instead of replacing one fixed target.

1. Open the Scrapy project directory that contains scrapy.cfg.

   $ cd /srv/catalog_demo

   Project-specific commands such as scrapy crawl and scrapy settings load the active settings module from this directory.

2. Define the FEEDS setting in settings.py with the destination file, export format, field order, and overwrite policy.

   FEEDS = {
       "output/products.jsonl": {
           "format": "jsonlines",
           "fields": ["name", "price", "url"],
           "overwrite": True,
           "store_empty": False,
       },
   }

   fields keeps the exported keys in a stable order instead of relying on item creation order.

   overwrite: True replaces any existing file at the same path each time the spider runs.

3. Read the resolved project setting before the crawl.

   $ scrapy settings --get FEEDS
   {"output/products.jsonl": {"format": "jsonlines", "fields": ["name", "price", "url"], "overwrite": true, "store_empty": false}}

   This confirms the active project is loading the expected feed definition before the spider starts.

4. Run the spider so Scrapy writes items through the configured feed exporter.

   $ scrapy crawl catalog
   ##### snipped #####
   2026-04-16 05:57:01 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (3 items) in: output/products.jsonl
   2026-04-16 05:57:01 [scrapy.core.engine] INFO: Spider closed (finished)

   The feed exporter reports the saved file path when the crawl finishes successfully.

5. Open the exported file to confirm the configured field order and one-item-per-line JSON Lines output.

   $ cat output/products.jsonl
   {"name": "Starter Plan", "price": "$29", "url": "https://catalog.example.com/products/starter-plan.html"}
   {"name": "Team Plan", "price": "$79", "url": "https://catalog.example.com/products/team-plan.html"}
   {"name": "Growth Plan", "price": "$129", "url": "https://catalog.example.com/products/growth-plan.html"}

   JSON Lines writes one complete JSON object per line, which is easier to append, stream, and diff than a single JSON array.

6. Count the saved records when a quick end-of-run verification is enough.

   $ wc -l output/products.jsonl
          3 output/products.jsonl

   The line total matches the number of exported items because JSON Lines stores one item on each line.

• Omit overwrite or set it to False when a local filesystem feed should append to the current file instead of replacing it.
• Keep store_empty: False when empty crawl runs should avoid creating a new empty export file.
• Use output/%(name)s-%(time)s.jsonl when each spider or crawl run needs a separate file instead of one reusable target.
• Use How to export a feed as JSON Lines in Scrapy when the workflow needs to focus on the JSON Lines format itself instead of the broader FEEDS configuration flow.