How to start a Spark Connect server

Spark Connect lets notebooks, IDEs, and applications submit DataFrame work to a separate Apache Spark driver instead of embedding the driver in the client process. Starting the server locally gives developers a small client/server Spark environment before exposing the endpoint through shared infrastructure.

The Apache Spark distribution provides sbin/start-connect-server.sh for the server process and sbin/stop-connect-server.sh for shutdown. Binding the first test to 127.0.0.1 keeps the gRPC endpoint local while using Spark Connect's default port, 15002.

The Python client environment needs the Spark Connect extras, including gRPC, Pandas, and PyArrow. Match the client package version to the Spark distribution version where possible so the remote session uses the expected protocol.

1. Open a terminal in the extracted Apache Spark directory.

   $ pwd
   /opt/spark

   The directory should contain the bin and sbin folders from an Apache Spark distribution, not only a Python virtual environment.

2. Install the Spark Connect client dependencies in the Python environment that will run the smoke test.

   $ python3 -m pip install "pyspark[connect]==4.1.2"

   Replace 4.1.2 with the Apache Spark version used by the server when validating a different release.

3. Start the Spark Connect server on the local interface.

   $ ./sbin/start-connect-server.sh \
     --conf spark.connect.grpc.binding.address=127.0.0.1 \
     --conf spark.connect.grpc.binding.port=15002
   starting org.apache.spark.sql.connect.service.SparkConnectServer, logging to /opt/spark/logs/spark-spark-org.apache.spark.sql.connect.service.SparkConnectServer-1-server.out

   Bind to a routable address only after network access, authentication, and transport security are planned for the environment.

4. Create a small remote-session smoke test.

   $ cat > connect_check.py <<'PY'
   from pyspark.sql import SparkSession
   
   spark = SparkSession.builder.remote("sc://localhost:15002").appName("sg-connect-check").getOrCreate()
   print("session_class=" + type(spark).__module__ + "." + type(spark).__name__)
   print("range_count=" + str(spark.range(5).count()))
   spark.stop()
   PY

5. Run the smoke test through the Spark Connect endpoint.

   $ python3 connect_check.py
   session_class=pyspark.sql.connect.session.SparkSession
   range_count=5

   The session class should include pyspark.sql.connect, which confirms the client used Spark Connect instead of a classic embedded SparkSession.

6. Remove the smoke-test script after the remote session works.

   $ rm connect_check.py

   Run ./sbin/stop-connect-server.sh later when the Spark Connect server should stop accepting client sessions.