How to read and write ORC files with Spark

ORC stores columnar data for large analytical workloads, especially in Hadoop, Hive, and data lake environments. Apache Spark can read and write ORC through its built-in SQL data source, so a PySpark job can move filtered DataFrames into an ORC directory without leaving the DataFrame API.

The PySpark reader accepts an ORC directory path and returns a DataFrame. The writer saves a DataFrame as a directory of part files, with optional settings such as compression and partition columns applied during the write. Point downstream Spark readers at the output directory, not at one individual part file.

A local-mode smoke test is enough to prove the handoff before the same pattern is moved into a submitted batch job. The job writes a small paid-order dataset with Snappy compression and region partition folders, reads the directory back, and checks that the expected rows and partition values are visible.

1. Create the PySpark ORC read/write job.

   orc_read_write.py

   from pyspark.sql import SparkSession
   from pyspark.sql.functions import col
   from pyspark.sql.types import (
       DoubleType,
       StringType,
       StructField,
       StructType,
   )
    
   spark = (
       SparkSession.builder
       .appName("sg-orc-read-write")
       .master("local[*]")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   schema = StructType([
       StructField("order_id", StringType(), False),
       StructField("region", StringType(), False),
       StructField("amount", DoubleType(), False),
       StructField("status", StringType(), False),
       StructField("order_date", StringType(), False),
   ])
    
   orders = spark.createDataFrame([
       ("ord-1001", "emea", 149.50, "paid", "2026-07-07"),
       ("ord-1002", "na", 87.25, "paid", "2026-07-07"),
       ("ord-1003", "emea", 42.00, "cancelled", "2026-07-07"),
   ], schema)
    
   paid_orders = (
       orders
       .where(col("status") == "paid")
       .select("order_id", "region", "amount", "order_date")
   )
    
   print("Input row count:", orders.count())
   orders.printSchema()
   paid_orders.show(truncate=False)
    
   (
       paid_orders
       .coalesce(1)
       .write
       .mode("overwrite")
       .option("compression", "snappy")
       .partitionBy("region")
       .orc("output/orc-sales")
   )
    
   read_back = spark.read.orc("output/orc-sales").orderBy("order_id")
    
   print("Read-back row count:", read_back.count())
   read_back.printSchema()
   read_back.show(truncate=False)
    
   spark.stop()

   coalesce(1) keeps the tiny sample to one part file per partition folder. Let production jobs write the number of part files chosen by the cluster unless a downstream handoff requires a smaller file count.

2. Run the Spark job.

   $ spark-submit orc_read_write.py
   ##### snipped #####
   Input row count: 3
   root
    |-- order_id: string (nullable = false)
    |-- region: string (nullable = false)
    |-- amount: double (nullable = false)
    |-- status: string (nullable = false)
    |-- order_date: string (nullable = false)
   
   +--------+------+------+----------+
   |order_id|region|amount|order_date|
   +--------+------+------+----------+
   |ord-1001|emea  |149.5 |2026-07-07|
   |ord-1002|na    |87.25 |2026-07-07|
   +--------+------+------+----------+
   
   Read-back row count: 2
   root
    |-- order_id: string (nullable = true)
    |-- amount: double (nullable = true)
    |-- order_date: string (nullable = true)
    |-- region: string (nullable = true)
   
   +--------+------+----------+------+
   |order_id|amount|order_date|region|
   +--------+------+----------+------+
   |ord-1001|149.5 |2026-07-07|emea  |
   |ord-1002|87.25 |2026-07-07|na    |
   +--------+------+----------+------+

   The read-back schema includes the region partition column even though Spark stored it in folder names. The row count and sample values confirm that the ORC directory was written and read back through Spark.

3. Inspect the ORC output directory.

   $ ls -R output/orc-sales
   output/orc-sales:
   _SUCCESS
   region=emea
   region=na
   
   output/orc-sales/region=emea:
   part-00000-e1b1862b-ac68-4edb-85fa-92af11b364a7.c000.snappy.orc
   
   output/orc-sales/region=na:
   part-00000-e1b1862b-ac68-4edb-85fa-92af11b364a7.c000.snappy.orc

   _SUCCESS marks a completed write. The region=<value> folders are the partition layout, and the part-*.snappy.orc files contain the ORC data.

4. Remove the local sample files after the read-back check passes.

   $ rm -r orc_read_write.py output