How to export Scrapy items to JSON

Exporting Scrapy items to JSON saves the crawl result as one structured file that other tools can archive, diff, or load without an extra conversion step.

Running scrapy crawl with -O and a target name ending in .json uses Scrapy's feed export system to write a JSON array. Each yielded item is serialized during the crawl, and the closing bracket is written when the crawl finishes successfully.

JSON works well for snapshot-style exports, but it is not a good append format. Current Scrapy documentation still warns that appending to an existing JSON file with -o makes the result invalid JSON, so repeated or large exports are usually better served by JSON Lines instead.

1. Open a terminal in the Scrapy project directory.

   $ cd /srv/catalog_demo

   Run the command from the directory that contains scrapy.cfg so Scrapy loads the correct project settings and spider names.

2. Run the spider with -O and a .json output file.

   $ scrapy crawl catalog -O products.json
   ##### snipped #####
   2026-04-16 05:40:17 [scrapy.extensions.feedexport] INFO: Stored json feed (3 items) in: products.json

   The file extension selects the exporter automatically, so products.json uses Scrapy's built-in JSON feed exporter.

   -O overwrites any existing products.json in place.

3. Open the exported file to confirm the crawl wrote one valid JSON array.

   $ cat products.json
   [
   {"name": "Starter Plan", "price": "$29", "url": "https://example.com/starter-plan"},
   {"name": "Team Plan", "price": "$79", "url": "https://example.com/team-plan"},
   {"name": "Growth Plan", "price": "$129", "url": "https://example.com/growth-plan"}
   ]

   The closing ] appears when the crawl finishes. If the crawl stops early, the file can be left as a truncated JSON array.

4. Parse the file once before handing it off to another tool.

   $ python3 -c "import json; json.load(open('products.json')); print('OK')"
   OK

   A successful parse confirms the export is complete and readable by another JSON-aware tool.

• Use -O when the JSON file should be replaced by the latest crawl result.
• Avoid -o products.json for repeat runs. Scrapy's current documentation still notes that appending to a JSON file produces invalid JSON.
• Set FEED_EXPORT_FIELDS in settings.py when downstream tools or diffs need a predictable key order in each exported item.
• Use How to export a feed as JSON Lines in Scrapy when the export must be appended between runs or processed record by record.