How to troubleshoot Spark out-of-memory errors

Spark out-of-memory failures usually come from one process crossing one memory boundary, not from the whole application simply needing a larger cluster. The fastest fix starts by matching the exception to the driver, executor, Python worker, or cluster container that ran out of room.

Driver heap failures often appear when a job returns too much data to the client with actions such as collect(). Executor heap failures usually appear in task logs during joins, shuffles, aggregations, or cached data reuse. Cluster-manager messages about memory overhead point outside the JVM heap, which is a different setting from spark.executor.memory or spark.driver.memory.

Use the smallest reproduction that still shows the original failure, then change the setting or code path tied to that signal. Raising every memory value at once can hide the real data-shape problem and make the next failure more expensive to debug.

1. Reproduce the failing action and keep the first out-of-memory line from the driver or executor log.

   $ spark-submit --master local[2] --driver-memory 512m --conf spark.driver.maxResultSize=0 driver_collect_oom.py
   Traceback (most recent call last):
     File "driver_collect_oom.py", line 13, in <module>
       rows.collect()
   py4j.protocol.Py4JJavaError: An error occurred while calling o35.collectToPython.
   : java.lang.OutOfMemoryError: Java heap space
   	at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:462)
   ##### snipped #####

   A driver-side collectToPython or executeCollect frame means data was being returned to the driver process. An executor-side stack trace, lost executor message, or cluster container-kill message needs executor or overhead settings instead.

2. Classify the memory boundary before changing settings.

   Java heap space in the driver log points to spark.driver.memory, spark.driver.maxResultSize, or a driver action that returns too much data. The same heap error in executor logs points to spark.executor.memory. Memory Overhead Exceeded, container killed by YARN, and Kubernetes pod memory kills point to spark.driver.memoryOverhead or spark.executor.memoryOverhead. Python worker RSS growth points to spark.executor.pyspark.memory when that limit is configured, or to executor memory overhead when it is not.

3. Confirm the active settings from the failed submission record before rerunning.

   The Spark UI Environment tab lists properties explicitly set by spark-submit, SparkConf, or spark-defaults.conf. Missing memory values normally mean the documented default is active, so preserve the failed submit command or cluster application details before changing multiple knobs.

   Related: How to check Spark application status

4. Replace a full driver collect with a bounded action when the driver heap is the failing boundary.

   rows = spark.range(0, 12000000, 1, 32).selectExpr("CAST(id AS STRING) AS payload")
   print("row_count =", rows.count())
   print("sample =", [row.payload for row in rows.limit(3).collect()])

   Do not set spark.driver.maxResultSize to 0 just to force a large collect() through the driver. That removes the result-size guard and can turn a controlled failure into a driver heap crash.

5. Rerun the workload with the setting tied to the classified boundary.

   $ spark-submit --master local[2] --driver-memory 2g --conf spark.driver.maxResultSize=1g driver_collect_fixed.py
   row_count = 12000000
   sample = ['0', '1', '2']

   For executor heap failures, change --executor-memory or spark.executor.memory instead of the driver heap. For YARN or Kubernetes memory-overhead failures, increase the matching memoryOverhead setting and retest the same input.
   Related: How to configure Spark executor memory
   Related: How to set Spark SQL shuffle partitions