How to scrape an XML file with Scrapy

Many catalog and partner feeds already expose the records a crawler needs as XML, without the layout noise and selector drift of an HTML page. Scrapy can stream each repeated element into one item while preserving the source's structured fields.

Scrapy's XMLFeedSpider handles this data flow through itertag and parse_node(). The tag selects each repeated record, while the callback receives a selector scoped to that element; the default iternodes iterator avoids loading the complete document tree for large feeds.

The feed URL must return the XML document itself, and its field paths must match any namespaces in the source. A zero-item export can therefore mean a wrong itertag or namespace mapping even when the request succeeded, so inspecting the response first separates transport problems from selector problems.

1. Inspect the repeated product identifiers in the XML response.

   $ scrapy shell --nolog https://files.example.net/catalog/products.xml -c '(type(response).__name__, response.xpath("//product/@sku").getall())'
   ('XmlResponse', ['starter-001', 'team-001', 'growth-001'])

   Default XML namespaces require a registered prefix in Scrapy because an unprefixed XPath does not inherit the document's default namespace.
   Tool: XPath Tester

2. Create product_xml_spider.py with the feed identity and repeated product element.

   product_xml_spider.py

   from scrapy.spiders import XMLFeedSpider
    
    
   class ProductXmlSpider(XMLFeedSpider):
       name = "product_xml"
       start_urls = ["https://files.example.net/catalog/products.xml"]
       iterator = "iternodes"
       itertag = "product"

   Related: How to create a Scrapy spider

3. Add parse_node() beneath itertag to map one product element into one item.

   product_xml_spider.py

       def parse_node(self, response, node):
           yield {
               "sku": node.xpath("@sku").get(),
               "name": node.xpath("name/text()").get(),
               "price": node.xpath("price/text()").get(),
               "url": node.xpath("url/text()").get(),
           }

4. Review the completed product_xml_spider.py.

   $ cat product_xml_spider.py
   from scrapy.spiders import XMLFeedSpider
   
   
   class ProductXmlSpider(XMLFeedSpider):
       name = "product_xml"
       start_urls = ["https://files.example.net/catalog/products.xml"]
       iterator = "iternodes"
       itertag = "product"
   
       def parse_node(self, response, node):
           yield {
               "sku": node.xpath("@sku").get(),
               "name": node.xpath("name/text()").get(),
               "price": node.xpath("price/text()").get(),
               "url": node.xpath("url/text()").get(),
           }

5. Run the completed spider with an overwritten JSON Lines export.

   The -O option replaces an existing products.jsonl file.

   $ scrapy runspider product_xml_spider.py -O products.jsonl --loglevel INFO
   2026-07-17 15:05:41 [scrapy.utils.log] INFO: Scrapy 2.14.1 started (bot: scrapybot)
   ##### snipped #####
   2026-07-17 15:05:42 [scrapy.extensions.feedexport] INFO: Stored jsonl feed (3 items) in: products.jsonl
   2026-07-17 15:05:42 [scrapy.core.engine] INFO: Spider closed (finished)

6. Confirm that products.jsonl contains one item for each product in the XML feed.

   $ cat products.jsonl
   {"sku": "starter-001", "name": "Starter Plan", "price": "29.00", "url": "https://shop.example.com/products/starter-plan.html"}
   {"sku": "team-001", "name": "Team Plan", "price": "79.00", "url": "https://shop.example.com/products/team-plan.html"}
   {"sku": "growth-001", "name": "Growth Plan", "price": "129.00", "url": "https://shop.example.com/products/growth-plan.html"}