How to run PySpark locally

PySpark local mode runs a Spark driver and worker threads on the same machine, which makes it a quick way to prove that Python, Java, and Apache Spark can start together before a job moves to spark-submit or a cluster manager. A small SparkSession check catches missing dependencies, local binding problems, and broken Python environments without needing external storage or a remote cluster.

The local[2] master uses two local worker threads. That is enough to run a real Spark action while keeping the test independent of YARN, Kubernetes, Standalone, or Spark Connect services.

A short Python script can create the session, run a range DataFrame action, print the active Spark version and master, and stop the session cleanly. Spark may print startup warnings before the script output; the success lines are the local[2] master, the expected row count, and the final stopped state.

1. Choose the local Spark runtime values for the check.

   Master: local[2]
   Application name: sg-pyspark-local-check
   Proof action: spark.range(0, 1000).count()

   local[2] runs Spark locally with two worker threads. Use local[*] for quick experiments that should use all available local cores.

2. Save the local PySpark check as sg_pyspark_local_check.py.

   sg_pyspark_local_check.py

   from pyspark.sql import SparkSession
    
   spark = (
       SparkSession.builder
       .master("local[2]")
       .appName("sg-pyspark-local-check")
       .config("spark.ui.showConsoleProgress", "false")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   print(f"spark_version={spark.version}")
   print(f"master={spark.sparkContext.master}")
   print(f"app_name={spark.sparkContext.appName}")
   print(f"range_count={spark.range(0, 1000).count()}")
    
   spark.stop()
   print("spark_stopped=true")

   The script uses SparkSession.builder because SparkSession is the entry point for DataFrame work in PySpark.

3. Run the script with the Python interpreter that has PySpark installed.

   $ python3 sg_pyspark_local_check.py
   WARNING: Using incubator modules: jdk.incubator.vector
   Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
   Setting default log level to "WARN".
   ##### snipped #####
   spark_version=4.1.2
   master=local[2]
   app_name=sg-pyspark-local-check
   range_count=1000
   spark_stopped=true

   Startup log lines can vary by Java version and platform. The script output should still show master=local[2], range_count=1000, and spark_stopped=true. A Java error usually points to the local Java runtime, and a Python import error usually means the active interpreter does not have pyspark installed.

4. Remove the check script after the local run is proven.

   $ rm sg_pyspark_local_check.py