How to upgrade an Apache Spark runtime

Local Apache Spark runtimes often sit between application code and larger Spark deployments. A developer workstation, scheduled edge node, or CI runner may launch jobs with the same spark-submit and PySpark jars that production jobs expect. Recording the old runtime, saving the package set, and running a smoke job before the package change gives the upgrade a clear recovery point.

A Python virtual environment keeps the Spark launcher scripts, PySpark package, Py4J dependency, and matching Spark jars in one isolated runtime. Upgrading that environment leaves system Python and any host-wide Spark installation untouched while still testing Spark through the normal spark-submit launcher.

Version compatibility still matters before a major Spark jump. Spark 4.1 runs on Java 17 or 21 and Python 3.10 or newer, and Scala applications need libraries built for the Scala line used by the Spark release. Check the relevant Spark Core, SQL, PySpark, streaming, and connector migration notes before applying the same runtime change to shared nodes or clusters.

1. Activate the Spark virtual environment that owns the local runtime.

   $ source ~/spark-venv/bin/activate

   Use the environment that provides the spark-submit command used by the jobs being upgraded.

2. Record the current PySpark package version.

   (spark-venv) $ python -m pip show pyspark
   Name: pyspark
   Version: 4.0.3
   Summary: Apache Spark Python API
   ##### snipped #####
   Location: /home/sparkuser/spark-venv/lib/python3.12/site-packages
   Requires: py4j
   Required-by:

3. Save the current package set for rollback.

   (spark-venv) $ python -m pip freeze > ~/spark-venv/requirements.pre-upgrade.txt

   Reinstall this file with python -m pip install --requirement ~/spark-venv/requirements.pre-upgrade.txt if the upgraded smoke job fails. This restores Python packages only; it does not undo table, schema, checkpoint, or external storage changes.

4. Save a Spark smoke job that exercises the runtime through SparkSession.

   ~/spark-upgrade-smoke.py

   from pyspark.sql import SparkSession
    
   spark = (
       SparkSession.builder
       .master("local[2]")
       .appName("sg-upgrade-smoke")
       .config("spark.ui.showConsoleProgress", "false")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   rows = spark.range(1, 4).count()
   print(f"spark_version={spark.version}")
   print(f"rows={rows}")
    
   spark.stop()

   local[2] starts the driver and two local worker threads on the same host. Replace this file with an existing representative job when the runtime must prove a project-specific connector, library, or input path.

5. Run the smoke job before changing the runtime.

   (spark-venv) $ spark-submit ~/spark-upgrade-smoke.py
   ##### snipped #####
   spark_version=4.0.3
   rows=3

6. Upgrade PySpark to the approved target version.

   (spark-venv) $ python -m pip install --upgrade pyspark==4.1.2
   Collecting pyspark==4.1.2
   ##### snipped #####
   Successfully installed pyspark-4.1.2

   Use an exact version pin so every upgraded runtime resolves to the same Spark release.

7. Confirm that the virtual environment now resolves the target package.

   (spark-venv) $ python -m pip show pyspark
   Name: pyspark
   Version: 4.1.2
   Summary: Apache Spark Python API
   ##### snipped #####
   License: Apache-2.0
   Location: /home/sparkuser/spark-venv/lib/python3.12/site-packages
   Requires: py4j
   Required-by:

8. Run the same smoke job after the upgrade.

   (spark-venv) $ spark-submit ~/spark-upgrade-smoke.py
   ##### snipped #####
   spark_version=4.1.2
   rows=3

   The repeated row count and new Spark version show that the upgraded runtime can start SparkSession and complete a DataFrame action through spark-submit.