How to add packages to a Spark job

Spark jobs often need connector or data source libraries that are not bundled with the runtime. Adding a package at submit time keeps the cluster installation unchanged while giving the driver and executors the extra classes for one application run.

The --packages option resolves a Maven coordinate and its transitive dependencies before the application starts. Spark places the resolved JARs on the driver and executor classpaths, which makes the package available to DataFrame readers, writers, and JVM code during that submitted job.

The package check uses spark-avro because Spark documents it as an external module that is not included in spark-submit or spark-shell by default. Use --jars for staged JAR files and --py-files for Python modules; keep repository credentials out of submit commands and prefer an internal repository or secret-managed runtime configuration when a private artifact source is required.

1. Collect the package coordinate and submit scope.

   Package: org.apache.spark:spark-avro_2.13:4.1.2
   Runtime: Spark 4.1.2
   Master: local[2]
   Application name: sg-package-check

   Match Spark project artifacts to the runtime version and Scala binary suffix. Spark 4.1.2 uses the _2.13 Spark module suffix shown here.

2. Save the package-backed PySpark job as spark_avro_package_check.py.

   spark_avro_package_check.py

   import shutil
    
   from pyspark.sql import SparkSession
    
   base_path = "/tmp/sg-spark-package"
   avro_path = f"{base_path}/orders-avro"
   shutil.rmtree(base_path, ignore_errors=True)
    
   spark = (
       SparkSession.builder.master("local[2]")
       .appName("sg-package-check")
       .getOrCreate()
   )
   spark.sparkContext.setLogLevel("ERROR")
    
   orders = spark.createDataFrame(
       [
           ("ORD-1001", "APAC", 3),
           ("ORD-1002", "EMEA", 1),
       ],
       "order_id string, region string, item_count int",
   )
    
   orders.write.format("avro").mode("overwrite").save(avro_path)
   loaded = spark.read.format("avro").load(avro_path)
    
   print(f"avro_rows={loaded.count()}")
   loaded.orderBy("order_id").show(truncate=False)
    
   spark.stop()
   shutil.rmtree(base_path, ignore_errors=True)

   The script writes and reads Avro data. A missing package causes the Avro format lookup to fail before the row count prints.
   Related: How to read and write Avro files with Spark

3. Submit the job with the spark-avro package.

   $ spark-submit \
     --master local[2] \
     --name sg-package-check \
     --conf spark.ui.showConsoleProgress=false \
     --packages org.apache.spark:spark-avro_2.13:4.1.2 \
     spark_avro_package_check.py
   org.apache.spark#spark-avro_2.13 added as a dependency
   ##### snipped #####
   avro_rows=2
   +--------+------+----------+
   |order_id|region|item_count|
   +--------+------+----------+
   |ORD-1001|APAC  |3         |
   |ORD-1002|EMEA  |1         |
   +--------+------+----------+

   --packages accepts comma-separated Maven coordinates and resolves transitive dependencies. Add --repositories only when the package lives outside the default repositories, and avoid embedding private repository passwords in the URI. Use --jars instead when the dependency is already staged as a JAR file; local:///opt/spark/jars/custom-source.jar means the file already exists at that path on every worker node.

4. Remove the temporary package-check script.

   $ rm spark_avro_package_check.py