How to read and write JSON files with Spark

JSON logs, events, and exports often reach Apache Spark as one record per line. Reading and writing those files with Spark lets a data job apply a schema, transform selected records, and hand a new JSON dataset to another batch job without leaving the DataFrame API.

PySpark local mode and the built-in JSON data source handle the file reads and writes here. An explicit schema makes the expected columns and types visible before the job writes anything. Spark can infer a schema, but inference reads the input once before the real work starts.

Spark writes JSON output as a directory of part files rather than a single file path. The small output is kept to one part file for inspection, then read back into a DataFrame so the row count and values prove the write succeeded before the temporary files are removed.

1. Create a newline-delimited JSON input file.

   $ cat > events.jsonl <<'JSON'
   {"event_id":"evt-1001","customer":"northwind","amount":19.95,"status":"placed","event_ts":"2026-07-07T08:15:00Z"}
   {"event_id":"evt-1002","customer":"contoso","amount":42.50,"status":"paid","event_ts":"2026-07-07T08:18:00Z"}
   {"event_id":"evt-1003","customer":"northwind","amount":7.25,"status":"cancelled","event_ts":"2026-07-07T08:21:00Z"}
   JSON

   Each nonblank line is its own complete JSON object. Use Spark's multiLine option only when each file contains one regular pretty-printed JSON document. For production input, validate the source as JSON Lines before loading it.
   Tool: JSON Validator

2. Create the PySpark read/write job.

   json_read_write.py

   from pyspark.sql import SparkSession
   from pyspark.sql.functions import col
   from pyspark.sql.types import DoubleType, StringType, StructField, StructType
    
   spark = (
       SparkSession.builder
       .appName("sg-json-read-write")
       .master("local[*]")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   schema = StructType([
       StructField("event_id", StringType(), True),
       StructField("customer", StringType(), True),
       StructField("amount", DoubleType(), True),
       StructField("status", StringType(), True),
       StructField("event_ts", StringType(), True),
   ])
    
   events = spark.read.schema(schema).json("events.jsonl")
   paid_events = (
       events
       .where(col("status") == "paid")
       .select("event_id", "customer", "amount", "event_ts")
   )
    
   print("Input row count:", events.count())
   events.printSchema()
   paid_events.show(truncate=False)
    
   paid_events.coalesce(1).write.mode("overwrite").json("output/json-events")
   read_back = spark.read.schema(paid_events.schema).json("output/json-events")
    
   print("Output row count:", read_back.count())
   read_back.show(truncate=False)
    
   spark.stop()

   coalesce(1) keeps the tiny dataset to one part file so the result is easy to inspect. Let production jobs write multiple part files unless a downstream handoff truly requires one file.

3. Run the Spark job.

   $ spark-submit json_read_write.py
   ##### snipped #####
   Input row count: 3
   root
    |-- event_id: string (nullable = true)
    |-- customer: string (nullable = true)
    |-- amount: double (nullable = true)
    |-- status: string (nullable = true)
    |-- event_ts: string (nullable = true)
   
   +--------+--------+------+--------------------+
   |event_id|customer|amount|event_ts            |
   +--------+--------+------+--------------------+
   |evt-1002|contoso |42.5  |2026-07-07T08:18:00Z|
   +--------+--------+------+--------------------+
   
   Output row count: 1
   +--------+--------+------+--------------------+
   |event_id|customer|amount|event_ts            |
   +--------+--------+------+--------------------+
   |evt-1002|contoso |42.5  |2026-07-07T08:18:00Z|
   +--------+--------+------+--------------------+

   The first table is the filtered DataFrame before writing. The second table comes from reading output/json-events back after Spark writes the directory.

4. Inspect the JSON part file Spark wrote.

   $ cat output/json-events/part-*.json
   {"event_id":"evt-1002","customer":"contoso","amount":42.5,"event_ts":"2026-07-07T08:18:00Z"}

   The file remains newline-delimited JSON. A larger job normally writes several part-* files under the output directory.

5. Remove the sample files after the verification check.

   $ rm -r events.jsonl json_read_write.py output