How to write partitioned data with Spark

Partitioned file output lets Apache Spark store rows in folders named from column values, such as region=apac/order_date=2026-07-07. This layout is useful when later Spark SQL or Hive-style readers commonly filter by those same columns, because Spark can discover partition values from the path and avoid scanning unrelated folders.

DataFrameWriter.partitionBy() belongs to the write path, not the in-memory partition count. It creates a Hive-style directory layout for file-based sinks such as Parquet, JSON, CSV, and ORC while the output still remains one dataset rooted at the directory passed to the writer.

Choose partition columns with values that group the data at a useful query level. A high-cardinality column can create many small directories or files, and the local check uses coalesce(1) only to keep the sample output compact.

1. Create the PySpark partitioned-write job.

   partitioned_write_check.py

   from pathlib import Path
   import shutil
    
   from pyspark.sql import SparkSession
   from pyspark.sql.functions import col
   from pyspark.sql.types import DoubleType, StringType, StructField, StructType
    
   spark = (
       SparkSession.builder
       .appName("sg-dataframe-write-partitioned")
       .master("local[2]")
       .config("spark.ui.enabled", "false")
       .config("spark.ui.showConsoleProgress", "false")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   base_path = Path("spark-partitioned-demo")
   output = base_path / "output" / "orders"
   shutil.rmtree(base_path, ignore_errors=True)
    
   schema = StructType(
       [
           StructField("order_id", StringType(), False),
           StructField("region", StringType(), False),
           StructField("order_date", StringType(), False),
           StructField("status", StringType(), False),
           StructField("amount", DoubleType(), False),
       ]
   )
    
   orders = spark.createDataFrame(
       [
           ("ord-1001", "apac", "2026-07-07", "paid", 212.10),
           ("ord-1002", "emea", "2026-07-07", "paid", 149.50),
           ("ord-1003", "emea", "2026-07-08", "cancelled", 42.00),
           ("ord-1004", "na", "2026-07-08", "paid", 87.25),
       ],
       schema,
   )
    
   (
       orders
       .coalesce(1)
       .write
       .mode("overwrite")
       .partitionBy("region", "order_date")
       .parquet(str(output))
   )
    
   partition_dirs = sorted(
       str(path.relative_to(output))
       for path in output.glob("region=*/order_date=*")
       if path.is_dir()
   )
    
   print("Partition directories:")
   for directory in partition_dirs:
       print(directory)
    
   read_back = spark.read.parquet(str(output))
   print("Read-back row count:", read_back.count())
   read_back.orderBy("order_id").show(truncate=False)
    
   apac_paid = read_back.where(
       (col("region") == "apac") & (col("order_date") == "2026-07-07")
   )
   print("Filtered row count:", apac_paid.count())
   apac_paid.select("order_id", "region", "order_date", "amount").show(truncate=False)
    
   spark.stop()

   mode(“overwrite”) replaces the sample output directory. Use append only when the destination layout, schema, and duplicate-handling rules are already controlled by the job.

2. Run the job in local Spark.

   $ spark-submit --master 'local[2]' partitioned_write_check.py
   ##### snipped #####
   Partition directories:
   region=apac/order_date=2026-07-07
   region=emea/order_date=2026-07-07
   region=emea/order_date=2026-07-08
   region=na/order_date=2026-07-08
   Read-back row count: 4
   +--------+---------+------+------+----------+
   |order_id|status   |amount|region|order_date|
   +--------+---------+------+------+----------+
   |ord-1001|paid     |212.1 |apac  |2026-07-07|
   |ord-1002|paid     |149.5 |emea  |2026-07-07|
   |ord-1003|cancelled|42.0  |emea  |2026-07-08|
   |ord-1004|paid     |87.25 |na    |2026-07-08|
   +--------+---------+------+------+----------+
   
   Filtered row count: 1
   +--------+------+----------+------+
   |order_id|region|order_date|amount|
   +--------+------+----------+------+
   |ord-1001|apac  |2026-07-07|212.1 |
   +--------+------+----------+------+

   The read-back DataFrame gets region and order_date from the partition folders when it reads the dataset root.

3. Inspect the partitioned output directory.

   $ ls -R spark-partitioned-demo/output/orders
   spark-partitioned-demo/output/orders:
   _SUCCESS
   region=apac
   region=emea
   region=na
   
   spark-partitioned-demo/output/orders/region=apac:
   order_date=2026-07-07
   
   spark-partitioned-demo/output/orders/region=apac/order_date=2026-07-07:
   part-00000-6b8185b6-c0cc-44d1-ba41-79ba885e4614.c000.snappy.parquet
   
   spark-partitioned-demo/output/orders/region=emea:
   order_date=2026-07-07
   order_date=2026-07-08
   
   spark-partitioned-demo/output/orders/region=emea/order_date=2026-07-07:
   part-00000-6b8185b6-c0cc-44d1-ba41-79ba885e4614.c000.snappy.parquet
   
   spark-partitioned-demo/output/orders/region=emea/order_date=2026-07-08:
   part-00000-6b8185b6-c0cc-44d1-ba41-79ba885e4614.c000.snappy.parquet
   
   spark-partitioned-demo/output/orders/region=na:
   order_date=2026-07-08
   
   spark-partitioned-demo/output/orders/region=na/order_date=2026-07-08:
   part-00000-6b8185b6-c0cc-44d1-ba41-79ba885e4614.c000.snappy.parquet

   _SUCCESS marks the completed write. The nested region=value/order_date=value folders are the partition layout created by partitionBy(“region”, “order_date”).

4. Remove the local proof files.

   $ rm -r partitioned_write_check.py spark-partitioned-demo