How to configure Spark executor memory

Spark executor memory determines the JVM heap available to tasks after a cluster manager starts each executor. Sizing it per application helps joins, aggregations, caching, and Python workers run with enough room while keeping cluster capacity from being reserved by jobs that do not need it.

The main setting is spark.executor.memory, the heap for each executor JVM. Non-heap overhead stays separate through spark.executor.memoryOverhead, and PySpark jobs can also use spark.executor.pyspark.memory when Python worker memory needs its own limit.

Memory values must be in place before the SparkContext is created, usually through spark-submit or spark-defaults.conf. Command-line configuration overrides shared defaults, and the running application should show the active values in Spark logs, Spark configuration, or the cluster UI.

1. Choose the per-executor memory values for the application.

   Executor heap: 768m
   Memory overhead: 384m
   PySpark worker memory: 256m
   Local proof workers: 2 workers, 1 core each, 2048 MiB each

   spark.executor.memory sets executor JVM heap. Spark calculates spark.executor.memoryOverhead from an overhead factor when it is not set directly, with a 384 MiB minimum in current Spark 4 releases. Kubernetes non-JVM jobs use a higher default overhead factor than JVM jobs. Set spark.executor.pyspark.memory only when PySpark worker memory should have a separate limit.

2. Save a temporary Spark application that prints the active memory settings.

   $ vi executor_memory_check.py

3. Add the memory check code.

   executor_memory_check.py

   from pyspark.sql import SparkSession
    
   spark = SparkSession.builder.appName("sg-executor-memory-check").getOrCreate()
   spark.sparkContext.setLogLevel("ERROR")
    
   conf = spark.sparkContext.getConf()
    
   for key in (
       "spark.executor.memory",
       "spark.executor.memoryOverhead",
       "spark.executor.pyspark.memory",
   ):
       print(f"{key} = {conf.get(key, '<not set>')}")
    
   print(f"defaultParallelism = {spark.sparkContext.defaultParallelism}")
   print(f"row_count = {spark.range(0, 1000, 1, 2).count()}")
    
   spark.stop()

   The script reads the configuration after SparkContext starts and runs a small DataFrame action so executor scheduling is exercised.

4. Submit the application with executor memory settings.

   $ spark-submit \
     --master 'local-cluster[2,1,2048]' \
     --conf spark.executor.memory=768m \
     --conf spark.executor.memoryOverhead=384m \
     --conf spark.executor.pyspark.memory=256m \
     executor_memory_check.py
   INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(memoryOverhead -> name: memoryOverhead, amount: 384, memory -> name: memory, amount: 768, pyspark.memory -> name: pyspark.memory, amount: 256, offHeap -> name: offHeap, amount: 0)
   ##### snipped #####
   INFO StandaloneSchedulerBackend: Granted executor ID app-20260706212802-0000/0 with 1 core(s), 768.0 MiB RAM
   ##### snipped #####
   spark.executor.memory = 768m
   spark.executor.memoryOverhead = 384m
   spark.executor.pyspark.memory = 256m
   defaultParallelism = 2
   row_count = 1000

   local-cluster[2,1,2048] is a disposable local proof target. For production, replace it with the real --master value such as yarn, a spark:// master URL, or a k8s:// API URL, and use workload-sized memory values.

5. Confirm the Spark logs and application output show the intended values.

   The ResourceProfile line shows Spark built the executor resource profile with 768 MiB heap, 384 MiB overhead, and 256 MiB PySpark memory. The Granted executor line shows a worker accepted an executor with 768 MiB RAM, and row_count = 1000 confirms the application started and ran a Spark action.

6. Remove the temporary memory check application.

   $ rm executor_memory_check.py