How to explain a Spark DataFrame query plan

Spark DataFrame transformations form a logical query that Spark turns into a physical execution plan before work reaches executors. Reading that plan helps confirm where filters, aggregates, sorts, joins, and shuffles appear before spending time tuning the wrong part of a job.

The DataFrame.explain() method prints Spark's plan to the console. formatted mode is a readable first pass because it separates the physical outline from per-node detail, while extended mode shows the parsed, analyzed, optimized, and physical stages.

Plan IDs, generated attribute numbers, and adaptive execution markers can vary between Spark versions and settings. Match operator names and their order first, then compare exact IDs only when tracing a specific optimizer decision or execution plan change.

1. Create a small PySpark job that prints formatted and extended plans.

   dataframe_explain_plan.py

   from pyspark.sql import SparkSession
   from pyspark.sql import functions as F
    
    
   spark = (
       SparkSession.builder.appName("sg-dataframe-explain-plan")
       .master("local[2]")
       .config("spark.ui.enabled", "false")
       .config("spark.sql.adaptive.enabled", "true")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   orders = spark.createDataFrame(
       [
           ("EMEA", "analytics", 1200),
           ("EMEA", "analytics", 800),
           ("APAC", "platform", 500),
           ("EMEA", "platform", 700),
           ("AMER", "analytics", 400),
       ],
       ["region", "team", "amount"],
   )
    
   totals = (
       orders.filter(F.col("region") == "EMEA")
       .groupBy("team")
       .agg(F.sum("amount").alias("total_amount"))
       .orderBy(F.col("total_amount").desc())
   )
    
   print("Formatted plan")
   totals.explain(mode="formatted")
    
   print("Extended plan")
   totals.explain(mode="extended")
    
   print("Result rows")
   totals.show()
    
   spark.stop()

   formatted mode gives a physical-plan outline and node details. extended mode adds logical plan stages that show how Spark rewrites the query before choosing physical operators. Other modes include simple for only the physical plan, cost for statistics when available, and codegen for generated code when Spark can produce it.

   Related: How to create a Spark DataFrame
   Related: How to select and filter a Spark DataFrame

2. Run the job with a local Spark master.

   $ spark-submit --master 'local[2]' dataframe_explain_plan.py
   Formatted plan
   == Physical Plan ==
   AdaptiveSparkPlan (9)
   +- Sort (8)
      +- Exchange (7)
         +- HashAggregate (6)
            +- Exchange (5)
               +- HashAggregate (4)
                  +- Project (3)
                     +- Filter (2)
                        +- Scan ExistingRDD (1)
   
   ##### snipped #####

3. Check the formatted physical-plan outline for the operations created by the DataFrame transformation.

   Filter comes from the region == “EMEA” condition. HashAggregate comes from groupBy().agg(). Exchange shows repartitioning for aggregation and ordering. Sort comes from orderBy().

4. Use the extended output when the optimizer rewrite matters.

   Extended plan
   == Parsed Logical Plan ==
   'Sort ['total_amount DESC NULLS LAST], true
   +- Aggregate [team#1], [team#1, sum(amount#2L) AS total_amount#3L]
      +- Filter (region#0 = EMEA)
         +- LogicalRDD [region#0, team#1, amount#2L], false
   
   == Optimized Logical Plan ==
   Sort [total_amount#3L DESC NULLS LAST], true
   +- Aggregate [team#1], [team#1, sum(amount#2L) AS total_amount#3L]
      +- Project [team#1, amount#2L]
         +- Filter (isnotnull(region#0) AND (region#0 = EMEA))
            +- LogicalRDD [region#0, team#1, amount#2L], false
   
   == Physical Plan ==
   AdaptiveSparkPlan isFinalPlan=false
   +- Sort [total_amount#3L DESC NULLS LAST], true, 0
      +- Exchange rangepartitioning(total_amount#3L DESC NULLS LAST, 200), ENSURE_REQUIREMENTS, [plan_id=26]
         +- HashAggregate(keys=[team#1], functions=[sum(amount#2L)], output=[team#1, total_amount#3L])
   ##### snipped #####

   The optimized section adds isnotnull(region#0) before the equality filter, and the physical section shows the shuffle boundaries Spark will execute.

   Related: How to configure Spark adaptive query execution

5. Confirm the query result still matches the plan being inspected.

   Result rows
   +---------+------------+
   |     team|total_amount|
   +---------+------------+
   |analytics|        2000|
   | platform|         700|
   +---------+------------+

   explain() prints the plan. show() executes this small job so the inspected transformation has a concrete result.