How to install Apache Spark on Ubuntu or Debian

Apache Spark needs a supported JVM and a Spark runtime before local jobs can start on an APT-based workstation or server. On Ubuntu or Debian, APT is the right layer for Java and Python packaging tools, while PyPI provides the current PySpark package with the Spark Python API and command launchers.

Spark 4.1 runs on Java 17 or 21 and Python 3.10 or newer. The install path uses OpenJDK 21 explicitly so the runtime stays inside Spark's documented Java support range even when a distribution's default Java package points at a newer release.

A per-user Python virtual environment keeps PySpark, Py4J, and the generated Spark launcher scripts out of the system Python. After installation, spark-submit should report the Spark version, and a local SparkSession job should return a computed value without requiring YARN, Kubernetes, or a standalone Spark cluster.

1. Open a terminal with sudo privileges.
2. Refresh the APT package index.

   $ sudo apt update

3. Install Java 21 and the Python virtual environment tools.

   $ sudo apt install --assume-yes openjdk-21-jre-headless python3-venv python3-pip

   Use openjdk-17-jre-headless instead when the target Debian release does not package OpenJDK 21. Spark 4.1 documents Java 17 and 21 support.

4. Confirm the active Java runtime.

   $ java -version
   openjdk version "21.0.11" 2026-04-21
   OpenJDK Runtime Environment (build 21.0.11+10-1-26.04.2-Ubuntu)
   OpenJDK 64-Bit Server VM (build 21.0.11+10-1-26.04.2-Ubuntu, mixed mode, sharing)

5. Confirm the active Python runtime.

   $ python3 --version
   Python 3.14.4

   PySpark 4.1 requires Python 3.10 or newer. Use the packaged Python 3 runtime on current Ubuntu or Debian releases unless a project requires a different interpreter.

6. Create a Python virtual environment for Spark.

   $ python3 -m venv ~/spark-venv

7. Activate the virtual environment.

   $ source ~/spark-venv/bin/activate

   While the environment is active, python, pip, spark-submit, pyspark, and related launcher scripts resolve from ~/spark-venv/bin.

8. Upgrade pip inside the virtual environment.

   (spark-venv) $ python -m pip install --upgrade pip
   Requirement already satisfied: pip in ./spark-venv/lib/python3.14/site-packages
   ##### snipped #####
   Successfully installed pip-26.1.2

9. Install PySpark from PyPI.

   (spark-venv) $ python -m pip install pyspark
   Collecting pyspark
   ##### snipped #####
   Successfully installed py4j-0.10.9.9 pyspark-4.1.2

   Omit a version pin to install the current PyPI release. Use a command such as python -m pip install pyspark==4.1.2 when the local runtime must match a cluster, notebook image, or course environment.

10. Confirm the Spark launcher.

    (spark-venv) $ spark-submit --version
    WARNING: Using incubator modules: jdk.incubator.vector
    Welcome to
          ____              __
         / __/__  ___ _____/ /__
        _\ \/ _ \/ _ `/ __/  '_/
       /___/ .__/\_,_/_/ /_/\_\   version 4.1.2
          /_/
    Using Scala version 2.13.17, OpenJDK 64-Bit Server VM, 21.0.11
    ##### snipped #####

11. Save a temporary local Spark smoke test.

    /tmp/sg-spark-install-smoke.py

    from pyspark.sql import SparkSession
     
    spark = (
        SparkSession.builder
        .master("local[2]")
        .appName("sg-local-check")
        .getOrCreate()
    )
     
    result = spark.range(1, 6).selectExpr("sum(id) AS total").collect()[0]["total"]
    print(f"local_sum={result}")
    print(f"spark_version={spark.version}")
    spark.stop()

    local[2] runs the driver and two worker threads on the same host. That is enough to prove Java, Python, PySpark, and the Spark launcher can work together before using a cluster manager.

12. Run the smoke test with spark-submit.

    (spark-venv) $ spark-submit --master local[2] /tmp/sg-spark-install-smoke.py
    WARNING: Using incubator modules: jdk.incubator.vector
    Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
    ##### snipped #####
    local_sum=15
    spark_version=4.1.2

    Startup log lines vary by Java version and platform. The important signals are the printed sum and the Spark version from the running SparkSession.

13. Remove the temporary smoke test.

    (spark-venv) $ rm /tmp/sg-spark-install-smoke.py