How to read and write Parquet files with Spark

Parquet files are a common handoff format for Apache Spark jobs because they keep column data and schema metadata together in a directory of part files. Reading and writing Parquet with Spark lets a data job filter or reshape an analytical dataset while staying inside the DataFrame API.

PySpark local mode is enough to prove the file-format workflow before the same job moves to a cluster. The sample job creates a small source Parquet directory only so the read step is repeatable, then loads that directory with spark.read.parquet() and writes the filtered DataFrame back with DataFrameWriter.parquet().

Spark treats a Parquet dataset as a directory, not a single output file. The read-back check points Spark at the output directory, confirms the row count and sample values, and shows the partition folders plus part-*.snappy.parquet files that downstream readers should use as one dataset.

1. Create the PySpark Parquet read/write job.

   parquet_read_write.py

   from pathlib import Path
   from shutil import rmtree
    
   from pyspark.sql import SparkSession
   from pyspark.sql.functions import col
   from pyspark.sql.types import (
       DoubleType,
       StringType,
       StructField,
       StructType,
   )
    
   spark = (
       SparkSession.builder
       .appName("sg-parquet-read-write")
       .master("local[*]")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   base_path = Path("spark-parquet-demo")
   source_path = base_path / "input" / "orders"
   output_path = base_path / "output" / "paid-orders"
    
   if base_path.exists():
       rmtree(base_path)
    
   schema = StructType([
       StructField("order_id", StringType(), False),
       StructField("region", StringType(), False),
       StructField("amount", DoubleType(), False),
       StructField("status", StringType(), False),
       StructField("order_date", StringType(), False),
   ])
    
   seed_orders = spark.createDataFrame([
       ("ord-1001", "emea", 149.50, "paid", "2026-07-07"),
       ("ord-1002", "na", 87.25, "paid", "2026-07-07"),
       ("ord-1003", "emea", 42.00, "cancelled", "2026-07-07"),
       ("ord-1004", "apac", 212.10, "paid", "2026-07-08"),
   ], schema)
    
   (
       seed_orders
       .write
       .mode("overwrite")
       .partitionBy("region")
       .parquet(str(source_path))
   )
    
   orders = spark.read.parquet(str(source_path))
   paid_orders = (
       orders
       .where(col("status") == "paid")
       .select("order_id", "region", "amount", "order_date")
   )
    
   print("Source row count:", orders.count())
   orders.printSchema()
   paid_orders.orderBy("order_id").show(truncate=False)
    
   (
       paid_orders
       .coalesce(1)
       .write
       .mode("overwrite")
       .option("compression", "snappy")
       .partitionBy("region")
       .parquet(str(output_path))
   )
    
   read_back = spark.read.parquet(str(output_path)).orderBy("order_id")
    
   print("Read-back row count:", read_back.count())
   read_back.printSchema()
   read_back.show(truncate=False)
    
   spark.stop()

   The first write creates a local source dataset for a repeatable smoke test. In production, point spark.read.parquet() at the existing Parquet directory instead.

2. Run the Spark job.

   $ spark-submit parquet_read_write.py
   ##### snipped #####
   Source row count: 4
   root
    |-- order_id: string (nullable = true)
    |-- amount: double (nullable = true)
    |-- status: string (nullable = true)
    |-- order_date: string (nullable = true)
    |-- region: string (nullable = true)
   
   +--------+------+------+----------+
   |order_id|region|amount|order_date|
   +--------+------+------+----------+
   |ord-1001|emea  |149.5 |2026-07-07|
   |ord-1002|na    |87.25 |2026-07-07|
   |ord-1004|apac  |212.1 |2026-07-08|
   +--------+------+------+----------+
   
   Read-back row count: 3
   root
    |-- order_id: string (nullable = true)
    |-- amount: double (nullable = true)
    |-- order_date: string (nullable = true)
    |-- region: string (nullable = true)
   
   +--------+------+----------+------+
   |order_id|amount|order_date|region|
   +--------+------+----------+------+
   |ord-1001|149.5 |2026-07-07|emea  |
   |ord-1002|87.25 |2026-07-07|na    |
   |ord-1004|212.1 |2026-07-08|apac  |
   +--------+------+----------+------+

   Spark preserves Parquet schema metadata, but columns read from Parquet appear nullable for compatibility. The region column comes from partition folders during the read.

3. Inspect the Parquet output directory.

   $ ls -R spark-parquet-demo/output/paid-orders
   spark-parquet-demo/output/paid-orders:
   _SUCCESS
   region=apac
   region=emea
   region=na
   
   spark-parquet-demo/output/paid-orders/region=apac:
   part-00000-ca5f1b5c-d521-402b-9b0c-7669085325bf.c000.snappy.parquet
   
   spark-parquet-demo/output/paid-orders/region=emea:
   part-00000-ca5f1b5c-d521-402b-9b0c-7669085325bf.c000.snappy.parquet
   
   spark-parquet-demo/output/paid-orders/region=na:
   part-00000-ca5f1b5c-d521-402b-9b0c-7669085325bf.c000.snappy.parquet

   _SUCCESS marks a completed Spark write. The region=<value> folders are the partition layout, and each part-*.snappy.parquet file stores data for that partition.

4. Remove the local sample files after the read-back check passes.

   $ rm -r parquet_read_write.py spark-parquet-demo